Martin Loitzl's Blog

Lose a Node, Keep the Service: Building a Zero-Downtime 3-Node k3s Cluster

2026-05-16T15:00:00.000Z

There’s a specific kind of peace that comes from watching a node go dark and seeing your application not even flinch.

No alerts. No 500s. No panicked SSH sessions. Just a quiet log line somewhere saying a pod rescheduled, and life goes on.

This post is about building exactly that: a 3-node k3s cluster running VinylVault — a full-stack app backed by a MongoDB replica set, a Typesense Raft cluster, and stateless Node.js services — where losing any single node is a non-event.

The hardware is three Raspberry Pi 5s, each booting from a 1 TB M.2 NVMe SSD. The principles apply to any multi-node k3s setup.

The Threat Model

“High availability” gets thrown around loosely. Let’s be concrete. Our goal is:

Losing any one of three nodes — permanently or temporarily — must not cause application downtime.

That means every layer of the stack needs to survive a 1-of-3 failure:

Layer	Technology	Survival mechanism
Control plane	k3s + embedded etcd	Raft quorum: 2/3 nodes
Storage	Longhorn	3-replica volumes
Search index	Typesense	Raft cluster: 3 nodes
Database	MongoDB	Replica set: 1 primary + 2 secondaries
App pods	Deployment + HPA	Topology spread: 1 pod per node

Every layer uses the same fundamental idea: you need at least 2 surviving nodes to maintain quorum or keep a replica running. With 3 total nodes, losing 1 always leaves 2. That’s your margin.

Layer 1: k3s Control Plane with etcd

k3s ships with embedded etcd. In a 3-node server setup, all three nodes form an etcd cluster. Etcd requires ⌊N/2⌋ + 1 = 2 nodes for quorum, so losing 1 of 3 is fine.

The critical detail is the stable control-plane endpoint. If your API server address is a node IP and that node dies, kubectl breaks and new pods can’t register. You need a VIP.

We use kube-vip in ARP mode. It runs as a DaemonSet on control-plane nodes and floats a virtual IP across whichever node is currently active:

# kube-vip manifest injected into /var/lib/rancher/k3s/server/manifests/
# VIP: 192.168.1.50 — DNS: cloudforge1-api → 192.168.1.50

The k3s join command on all nodes always points at the VIP, never at a node IP:

K3S_URL=https://cloudforge1-api:6443

When a control-plane node fails, kube-vip re-elects and the VIP migrates. The API stays reachable. New pods can still be scheduled. The cluster continues.

Layer 2: Distributed Storage with Longhorn

Stateful services need persistent volumes. On a single node, a PVC is just a local directory. That’s fine until that node dies and the pod reschedules somewhere else — then the data is gone.

Longhorn solves this by replicating volumes across nodes. With defaultReplicaCount: 3, every PVC gets 3 replicas spread across your 3 nodes. The pod can reschedule to any surviving node and Longhorn will attach the volume there.

# gitops/clusters/homelab/platform/longhorn/helmrelease.yaml
values:
  defaultSettings:
    defaultReplicaCount: 3

A subtle catch: defaultReplicaCount only applies to new PVCs. Existing volumes keep their original replica count. If you started on a single node and then scaled to 3, you need to patch old volumes manually:

kubectl patch volume <pvc-name> -n longhorn-system \
  --type=merge -p '{"spec":{"numberOfReplicas":3}}'

Layer 3: MongoDB Replica Set

MongoDB’s replica set protocol is built for exactly this scenario. A 3-member set (1 primary, 2 secondaries) requires 2 nodes for election quorum. Lose 1 secondary: still have a primary. Lose the primary: the 2 remaining members elect a new one in seconds.

The non-obvious part is how you configure the member hostnames. This tripped us up.

When MongoDB joins a replica set, it registers each member’s host address and uses it for inter-member replication. If you initialize the set pointing member 0 at a Kubernetes ClusterIP service:

rs.initiate({ members: [{ _id: 0, host: "mongodb:27017" }] })

…you’ve made a trap. The mongodb ClusterIP will eventually load-balance across all 3 pods — including pods that are themselves in STARTUP state. When a new secondary tries to sync, it picks the ClusterIP as its source and randomly hits a non-primary. Sync fails, pod crashes, crashes loop.

Always use StatefulSet headless DNS for RS members:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongodb-0.mongodb-headless.vinylvault.svc.cluster.local:27017" },
    { _id: 1, host: "mongodb-1.mongodb-headless.vinylvault.svc.cluster.local:27017" },
    { _id: 2, host: "mongodb-2.mongodb-headless.vinylvault.svc.cluster.local:27017" },
  ]
})

With headless DNS, each mongodb-{N} name resolves directly to the pod’s IP. No load balancing. Replication always goes to the right pod.

We wrap this in a Kubernetes Job that runs on each fresh deploy. The init container waits until all 3 headless DNS names resolve before the main container runs rs.initiate() — making the job idempotent and crash-safe:

# init container: wait for all 3 members to be reachable
for host in mongodb-0.mongodb-headless mongodb-1.mongodb-headless mongodb-2.mongodb-headless; do
  until nc -z ${host}.${NAMESPACE}.svc.cluster.local 27017; do sleep 2; done
done

Layer 4: Typesense Raft Cluster

Typesense uses Raft for its internal cluster consensus. Like etcd and MongoDB, a 3-node cluster tolerates 1 failure.

The setup follows the same headless DNS pattern: each Typesense node needs to know the stable addresses of its peers at startup. We use an init container that resolves the headless DNS names and writes a nodes file:

mongodb-0.typesense-headless.vinylvault.svc.cluster.local:8107:8108,
mongodb-1.typesense-headless.vinylvault.svc.cluster.local:8107:8108,
mongodb-2.typesense-headless.vinylvault.svc.cluster.local:8107:8108

One thing that makes Typesense tricky in Kubernetes: pod IPs change on reschedule, but Typesense embeds peer addresses in its Raft log. To handle this gracefully, we run a nodes-updater sidecar that rewrites the nodes file every 20 seconds:

- name: nodes-updater
  image: alpine
  command: ["/bin/sh", "-c"]
  args:
    - |
      while true; do
        # resolve current IPs for all peers and rewrite /data/nodes
        ...
        sleep 20
      done

This means Typesense always has fresh peer addresses without needing a full restart after a pod reschedule.

Layer 5: Stateless Pods — the Topology Spread Trap

MongoDB and Typesense handle their own resilience. Stateless pods (backend API, BFF, frontend) need a different approach: just make sure there’s always at least one replica on a surviving node.

The naive solution — replicas: 2 — is not enough. Without placement constraints, Kubernetes might schedule both pods on the same node. Lose that node, lose both pods.

The fix is topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector: {}
    matchLabelKeys:
      - app

DoNotSchedule is the key word. ScheduleAnyway (the default) is a hint — the scheduler tries to spread, but will stack pods on one node if it’s under pressure. DoNotSchedule is a hard rule: if spreading is impossible, the pod stays Pending. That’s the guarantee we want.

Pair this with minReplicas: 2 in the HPA:

# infra/k8s/overlays/homelab/kustomization.yaml
- target:
    kind: HorizontalPodAutoscaler
  patch: |-
    - op: replace
      path: /spec/minReplicas
      value: 2
    - op: replace
      path: /spec/maxReplicas
      value: 3  # HPA can scale to fill all 3 nodes under load

With 3 nodes and 2 replicas, the topology constraint places 1 pod on 2 different nodes. Lose either node: 1 replica survives, HPA immediately targets 2, a new pod schedules on the now-empty third node.

One gotcha with rolling updates: during a rolling update, the scheduler sees terminating old pods as “still occupying” their node. New pods can temporarily stack on the same node before the old ones finish terminating. After every topology-changing deployment, a kubectl rollout restart re-spreads the pods cleanly.

What Surviving Node Loss Actually Looks Like

With all layers in place, this is what happens when a node goes down:

etcd detects the node is unreachable and marks it unhealthy. The remaining 2 nodes maintain quorum.
kube-vip (if the downed node held the VIP) re-elects and migrates the VIP to a healthy node within ~2 seconds. kubectl keeps working.
Longhorn rebuilds volume replicas on the surviving nodes. The volume remains accessible throughout.
MongoDB — if the failed node held a secondary, the replica set continues with the surviving primary + secondary. If it held the primary, an election completes in 10–30 seconds, a new primary is elected, and writes resume.
Typesense — same as MongoDB. 2/3 nodes maintain Raft quorum.
Stateless pods — the scheduler detects the node is NotReady, evicts pods after tolerationSeconds (default 300s for node.kubernetes.io/not-ready). With topology spread, the surviving node already has a replica running and serving traffic throughout.

The only visible impact: if the pod on the downed node was handling an in-flight request, that request fails. The next request goes to the surviving replica. For most workloads, that’s not “downtime” — it’s a retry.

The Numbers

Scenario	MongoDB	Typesense	Stateless
Node 1 down	RS: 1P + 1S ✅	Raft: 2/3 ✅	1 replica survives ✅
Node 2 down	RS: 1P + 1S ✅	Raft: 2/3 ✅	1 replica survives ✅
Node 3 down	RS: 1P + 1S ✅	Raft: 2/3 ✅	1 replica survives ✅
2 nodes down	RS: no quorum ❌	Raft: no quorum ❌	0 replicas ❌

You can lose any one. You can’t lose two. That’s the deal with N=3.

Is This Overkill for a Homelab?

Maybe. But “homelab” doesn’t mean “I don’t care about uptime.” It means I’m learning on my own hardware with my own time. And the thing I’ve learned most clearly from this project is that resilience doesn’t come from hope — it comes from explicit, testable configuration.

Every piece of this setup is reproducible from a fresh flash of three 1 TB NVMe SSDs. Plug in a node, watch it join, watch Flux reconcile, watch the replica count go up. It’s infrastructure as a first-class concern.

The Raspberry Pi 5s run k3s. The code runs in Git. The cluster runs itself.

The full setup lives in cloudforge (the GitOps infra repo) and vinyl-vault (the app).

VinylVault is a personal record collection manager and architecture sandbox. Read more about the application layer in The Greenfield Sandbox.

The Greenfield Sandbox: Implementing Hard Tenant Isolation and Deterministic Data at Scale

2026-02-14T19:08:00.000Z

In over a decade of enterprise software development, I’ve noticed a recurring frustration: the more senior you become, the rarer it is to work on a truly “greenfield” project. Most of our day-to-day work involves navigating existing tech debt, respecting legacy migrations, or making incremental improvements to architectures decided years before we joined.

It is notoriously difficult to test large-scale architectural ideas—like physical tenant isolation or the Backend-for-Frontend (BFF) pattern—within a established corporate codebase. The risk to production is too high, and the friction of legacy systems is too great.

VinylVault is my response to that constraint. It started as a personal record collection manager, but quickly evolved into a laboratory where I could implement a high-scale SaaS architecture from the ground up, focusing on three enterprise-level pillars: Hard Data Isolation, Security by Design, and Deterministic Metadata Aggregation.

1. Physical Isolation: Why `tenant_id` Isn’t Enough

In the enterprise world, “Multi-Tenancy” is often handled logically via a column in a shared table. While this works, it creates a massive “leaky abstraction” risk and makes heavy users (the “noisy neighbor” problem) hard to manage.

For VinylVault, I implemented Physical Tenant Isolation.

Dynamic Database Routing

Every user or organization gets their own dedicated MongoDB database (e.g., vinylvault_user_123). My Domain Backend acts as a dynamic router. When a request hits the API, the getTenantDb connection manager performs a just-in-time lookup:

It verifies the tenantId from a claims-based JWT.
It checks an internal connection pool (a Map<string, Db>).
If a connection is stale or missing, it dynamically connects to the tenant’s specific sandbox.

This ensures that a coding error in a resolver can never result in data leakage between users—the database connection itself simply doesn’t have access to other tenants’ data.

The Indexing Challenge

Managing hundreds of databases means you can’t manually manage indexes. I built an Asynchronous Reconciliation Layer. The first time a tenant accesses the system, the backend automatically verifies and creates the necessary text indexes and atomic counter documents before resolving the first GraphQL query.

2. The Blended Scoring Engine: Achieving Determinism

One of the hardest problems in music data is the inconsistency of community-driven APIs like Discogs and MusicBrainz. If you scan a barcode and get three results for “Nirvana - Nevermind,” how do you programmatically select the “correct” one?

I built a Normalization and Weighted Scoring Pipeline that treats these external APIs as raw, untrusted data.

Heuristic Grouping: I use configurable affixes to strip non-semantic noise like (2022 Remaster) or [Bonus Tracks], allowing me to group disparate results into a single deterministic groupingKey.
Weighted Scoring: Every release is scored based on metadata completeness (tracklists, label info, high-res cover art). Vinyl formats receive a specific weight boost.
Double Dagger Delimiters: To manage composite IDs across different sources, I used the ‡ (double dagger) character (e.g., SOURCE‡ID). This avoids the collision issues common with colons or dashes found in external IDs.

This ensures Traceability. My API doesn’t just return a result; it returns the “why” behind the choice, exposing the score breakdown for every candidate.

3. The BFF Pattern: Security by Design

Security in modern web apps is often compromised by storing sensitive JWTs in localStorage. To solve this, I implemented a Backend-for-Frontend (BFF) layer.

JWT Hiding

The BFF handles the “dirty work” of identity. The frontend only ever deals with a secure, HTTP-only session cookie.

The BFF validates the session.
It identifies the user’s active tenant.
It generates a short-lived, internally-scoped JWT to talk to the Domain Backend.

This pattern, known as “Token Exchange,” means the frontend never sees the primary auth tokens, significantly reducing the surface area for XSS attacks.

4. Scaling the Frontend: Strict Relay & Atomic Design

After a decade of watching Redux stores grow into unmaintainable monsters, I refactored the VinylVault frontend to use Relay and a custom Atomic Design System.

Fragment Colocation

Relay is my choice for enterprise scale because it forces Strict Fragment Colocation. Every component (like RecordCard) defines exactly what data it needs via a GraphQL fragment. This creates a “Contract-First” UI where over-fetching is physically impossible.

“Render-as-you-Fetch”

I implemented the useQueryLoader pattern. In the collection view, the data fetch begins the millisecond a user initiates a navigation event. By the time the component finishes its first render, the data is already arriving from the BFF.

The “Silent” Tenant-Switch Bug

In a multi-database architecture, the frontend cache is a liability. If you switch from your Personal collection to an Org collection, Relay’s normalized store might still hold IDs from the previous tenant.

To fix this, I implemented a Manual Store Purge during the tenant switch mutation:

const switchTenant = useCallback(async (tenantId: string) => {
  // 1. BFF updaes the session and returns new context
  const data = await executeGraphQLMutation(query, { tenantId });
  
  // 2. CRITICAL: Wipe the Relay cache to prevent ID overlap between DBs
  RelayEnvironment.getStore().getSource().clear();

  // 3. Update React context and navigate
  setActiveTenant(data.activeTenant);
  navigate('/'); 
}, []);

The Lessons Learned

Building VinylVault on my own terms allowed me to prove that “Enterprise Grade” doesn’t have to mean “Slow and Heavy.” By using a Monorepo (pnpm workspaces), Stateless Backend Routing, and Strict GraphQL Contracts, I’ve built a system that is arguably more robust than many of the production environments I’ve inherited in the past.

It turns out that when you remove the constraints of legacy systems, you can build software that is not only scalable and secure but a joy to maintain.

Star it on GitHub: Vinyl Vault

Live Demo @ Vinyl Vault - Demo

The Stack Summary:

Architecture: 3-Tier (Frontend -> BFF -> Domain Backend)
Isolation: Database-per-tenant (MongoDB)
Fetching: Relay (Strict Fragment Colocation)
Language: TypeScript (Strict Mode)
Branding: Tailwind CSS with a custom Atomic UI Library

Variance and covariance explained with C#

2025-08-25T19:08:00.000Z

The other day a colleague asked me about the difference between variance and covariance in C#. I smiled, and realized I probably owed him an article.

Variance in programming has a reputation for being boring theory reserved for academic types and Stack Overflow answers from 2011. But it is not. It’s one of those topics that silently runs the type system you rely on every day. And once you grasp it, you’ll spot opportunities to write cleaner, more composable code.

The Gentle Theory Bit

Let’s start with crisp definitions:

Variance describes how subtype relationships behave when a type parameter is involved.
Covariance means “I can go upwards in the inheritance tree safely.” If Pig inherits Animal, then IEnumerable<Pig> can be treated as IEnumerable<Animal>.
Contravariance means the opposite direction. It allows safe handling of more general inputs: if Pig is an Animal, then IComparer<Animal> can also be an IComparer<Pig>.

Or, put differently:

out = covariance (outputs)
in = contravariance (inputs)

Here’s a neat mnemonic: the N in in belongs to CoNtravariance. The out keyword simply points to coVariance, as it describes output positions.

Two classic examples from .NET’s BCL should be familiar:

public interface IEnumerable<out T> { ... }
public interface IComparer<in T> { ... }

Why these choices?

IEnumerable<T> produces sequence elements, so it’s covariant.
IComparer<T> consumes elements to compare them, so it’s contravariant.

Our Animal Hierarchy

First, the characters:

public abstract class Animal
{
    public string Name { get; }
    protected Animal(string name) => Name = name;
    public abstract void Speak();
}

public class Pig : Animal
{
    public Pig(string name) : base(name) { }
    public override void Speak() => 
        Console.WriteLine($"{Name}: Oink!");
}

public class Horse : Animal
{
    public Horse(string name) : base(name) { }
    public override void Speak() => 
        Console.WriteLine($"{Name}: Neigh!");
}

var napoleon = new Pig("Napoleon");
var boxer = new Horse("Boxer");

Covariance With `out`

Covariance shines when you produce (return) something. The classic example:
IEnumerable<out T> is covariant. This means you can assign a collection of derived types to a collection of base types:

IEnumerable<Pig> pigs = new List<Pig> { napoleon };
IEnumerable<Animal> animals = pigs; // Works thanks to covariance

foreach (var animal in animals)
{
    animal.Speak();
}

Here the variance lets us consume the sequence as its base type. Napoleon still squeals as expected.

Async Streams (`IAsyncEnumerable<out T>`)

With async streams introduced in C# 8, the variance principle shines:

async IAsyncEnumerable<Dog> GetDogsAsync() { ... }

// Covariance lets you assign to a more general type:
IAsyncEnumerable<Animal> animals = GetDogsAsync();

Without covariance, this neat and type-safe upcasting wouldn’t be allowed.

Read-Only Collections

Read-only interfaces are typically covariant since they only expose data:

IReadOnlyList<Dog> dogs = ...
IReadOnlyList<Animal> animals = dogs; // perfectly valid due to covariance

IReadOnlyList<out T> can produce values but not consume them, so safe covariance applies.

Contrast with mutable collections:

IList<Animal> animals = new List<Dog>(); // Error: IList<T> is invariant
animals.Add(new Cat());                  // This would break type safety, so disallowed

Async LINQ Extensions

Async LINQ operators use variance too:

await foreach (Animal a in GetDogsAsync().OfType<Animal>())
{
    ...
}

Here, covariance in IAsyncEnumerable<out T> allows seamless upcasting during enumeration.

Contravariance With `in`

Contravariance is the mirror world. It occurs where a type parameter only appears as an input.
Imagine we have a farm-wide announcer whose sole job is to talk to animals:

public interface IAnnouncer<in T>
{
    void Announce(T animal);
}

public class LoudAnnouncer : IAnnouncer<Animal>
{
public void Announce(Animal animal)
    => Console.WriteLine($"Announcing: {animal.Name} is here!");
}

Now, thanks to contravariance, we can reuse IAnnouncer<Animal> even if the code expects a IAnnouncer<Pig>:

IAnnouncer<Pig> pigAnnouncer = new LoudAnnouncer();
pigAnnouncer.Announce(napoleon);

The in keyword makes this possible: an announcer that works for “all animals” can also be used where “only pigs” are requested.

Dependency Injection And Contravariance

Variance becomes especially powerful when combined with dependency injection (DI).

Imagine you have a service that requires an announcer just for pigs:

public class PigShow
{
    private readonly IAnnouncer<Pig> _announcer;

    public PigShow(IAnnouncer<Pig> announcer) 
    {
        _announcer = announcer;
    }

    public void Run(Pig pig) => 
        _announcer.Announce(pig);
}

You don’t want to write multiple announcers for every animal type. Instead, you register a single IAnnouncer<Animal> in DI:

services.AddSingleton<IAnnouncer<Animal>, LoudAnnouncer>();
services.AddTransient<PigShow>();

Now DI happily resolves IAnnouncer<Pig> requests using the IAnnouncer<Animal> implementation, because contravariance made that conversion legal.

This is both elegant and practical: a single implementation scales across multiple injected scenarios.

Comparers and Dependency Injection

Contravariance comes in handy when you consume values, for example with comparers:

class AnimalNameComparer : IComparer<Animal>
{
    public int Compare(Animal? x, Animal? y) =>
        string.Compare(x?.Name, y?.Name, StringComparison.Ordinal);
}

IComparer<Dog> dogComparer = new AnimalNameComparer();

This compiles because IComparer<T> is contravariant (in T): a comparer for Animal can also handle Dog. Handy for registering services in dependency injection containers for base types while resolving derived types.

The Classic Pitfalls

Variance doesn’t come for free:

Covariance (out) applies only when you return data, never when you need to put something inside.
Contravariance (in) works when consuming inputs, not producing outputs.

Why isn’t Everything Covariant or Contravariant?

Variance is only safe if the interface respects its output/input constraints. If an interface both consumes and produces T, the compiler forbids variance to prevent runtime type errors.

Example:

IList<Animal> animals = new List<Dog>(); // Compilation error

Allowing variance here would mean you could insert a Cat into a list of Dog, breaking runtime safety.

Arrays are covariant

C# allows this:

string[] strings = new string[] { "a", "b", "c" };
object[] objects = strings; // ✅ Allowed, because arrays are covariant

So an array of string can be treated as an array of object. That’s covariance: string → object (derived → base).

Why is this unsafe?

At runtime you can now do:

object[] objects = new string[2];
objects[0] = "hello";   // fine
objects[1] = 42;        // ❌ runtime error!

At compile time, the compiler thinks this is fine (42 is an object). But at runtime, the actual array is a string[], and 42 is not a string. So the runtime throws an ArrayTypeMismatchException. 👉 This means covariance of arrays breaks type safety.

Variance In JavaScript And TypeScript

In JavaScript, you don’t have variance. Everything is duck typing and vibes.
But in TypeScript, variance sneaks in through assignability rules:

interface Announcer<T> {
    announce(animal: T): void;
}

let pigAnnouncer: Announcer<Pig>;
let animalAnnouncer: Announcer<Animal>;

By default, TypeScript treats function parameters as bivariant (both co- and contravariant-ish), which is convenient but not always type-safe.
Strict mode (strictFunctionTypes) nudges it closer to C#’s model:

Return types are covariant – it’s safe for a Pig to be returned where Animal is expected.
Parameter types are contravariant – you can provide an Announcer<Animal> if someone asks for an Announcer<Pig>.

So the concepts map over neatly, but TypeScript hides most of the painful corners for you. Meanwhile, C# makes you explicitly sprinkle in and out keywords like salt on your generics.

Conclusion

Variance is one of those “once seen, can’t be unseen” aspects of type systems.

Covariance (out) lets you use derived collections as base collections.
Contravariance (in) lets you use general consumers in more specific contexts.
Dependency injection frameworks love contravariance.
Even JavaScript/TypeScript play in this space, albeit more loosely.

And in case you forget: just remember the N in in belongs to CoNtravariance.

Deleting deleted group connected SharePoint Online Sites

2025-04-10T11:11:11.000Z

When a SharePoint Online site was deleted, it might have been connected to a M365 group. In this case it cannot be removed from deleted sites.

First delete the group from Entra (Azure AD)

Using PnP Powershell command Get-PnPDeletedMicrosoft365Group it is possible to find the deleted M365 Group, and remove it with Remove-PnPDeletedMicrosoft365Group:


PS /Users/martin> Get-PnPDeletedMicrosoft365Group

DisplayName                               Id                                   HasTeam SiteUrl
-----------                               --                                   ------- -------
PnP Microsoft 365 library test with group e351a3ff-1595-4979-b588-88038f993cc6 False

Remove-PnPDeletedMicrosoft365Group -Identity e351a3ff-1595-4979-b588-88038f993cc6

Afterwards the permanetly delete button works:

Bonus Feature: Since we are already connected, we can also use Get-PnPTenantDeletedSite and Remove-PnPTenantDeletedSite to remove the site permanently.


PS /Users/martin> Get-PnPTenantDeletedSite

SiteId                 : b0d12530-9a8d-4173-874b-eb8001090fdf
Url                    : https://loitzl2.sharepoint.com/sites/pnpcoresdktestgroup2
Status                 : Recycled
DeletionTime           : 10.04.2025 18:37:19
DaysRemaining          : 92
StorageQuota           : 26214400
ResourceQuota          : 300

PS /Users/martin> Remove-PnPTenantDeletedSite -Force https://loitzl2.sharepoint.com/sites/pnpcoresdktestgroup

Getting an App Only access token for SharePoint REST APIs

2023-03-13T11:11:11.000Z

Objectives

Recently I was asked to access SharePoint Online APIs using App Only access. I always used libraries (like MSAL, or PnP Core) for that and they always use signed client assertions for acquiring bearer tokens.

This time I needed some deeper insights and analyzed the whole process on protocol level.

App Registration

As a pre-requisite we need an Azure Active Directory App registration with access to SharePoint. There are plenty of guides on the net for that, for example Setup Authentication.

In short, I used openssl to create self-signed certificates…

# Create the certificate
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365  
# Convert to pfx file
$ openssl pkcs12 -export -out certificate.pfx -inkey key.pem -in cert.pem

…and the PnP PowerShell cmdlet Register-PnPAzureADApp to create the Azure AD app registration.

Register-PnPAzureADApp `
  -ApplicationName TestApp1 `
  -Tenant <tenant>.onmicrosoft.com `
  -CertificatePath certificate.pfx `
  -CertificatePassword (ConvertTo-SecureString -String "<cert password>" -AsPlainText -Force) `
  -Username "<user>@<tenant>.onmicrosoft.com" `
  -Password (ConvertTo-SecureString -String "<password>" -AsPlainText -Force) `
  -SharePointApplicationPermissions Sites.Read.All

Getting an OAuth2 Token using a Client Assertion

The Request for getting the app only access token looks like this:

POST https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id=<client-id>
&grant_type=client_credentials
&scope=https://<tenant>.sharepoint.com/.default
&client_assertion=<signed jwt client assertion>
&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer

The tenant-id is the id of the Azure AD tenant, client-id the app id of the Azure AD App registration. The grant type is set to client_credentials.

The scope needs to be set to the SharePoint URL, followed by .default.

The client assertion type is set to jwt-bearer, and the client assertion itself a base64 jwt similar to…

{
  "iat": 1678548857,
  "exp": 1678552457,
  "iss": "791ff0ac-1f93-47fc-8629-6b20f360036f",
  "sub": "791ff0ac-1f93-47fc-8629-6b20f360036f",
  "aud": "https://login.microsoftonline.com/0a7ab993-afa3-4a29-9b0b-f64583a4d888/oauth2/token"
}

…signed with the private key of the certificate provided to the Azure AD App registration.

The jwt contains the following claims:

“iat”: Issued at timestamp
“exp”: Expires timestamp - time window, as short as possible
“iss”: Issuer - the App Id
“sub”: Subject Claim - also the App Id
“aud”: Audience - the token endpoint

This is quite hard to do with command line tools, fortunately there is a handy npm package that does all that in one step:

The `jwtgen` Tool

$ jwtgen -a RS256 \
         -p key.pem \
         -e 3600 \
         -c "iss=<client-id>" \
         -c "sub=<client-id>" \
         -c "aud=https://login.microsoftonline.com/<tenant-id>/oauth2/token"  \
         --headers "{\"typ\":\"JWT\",\"alg\":\"RS256\",\"x5t\":\"<base64-encoded-certificate-thumbprint>\"}" \
         -v

Note that the certificate thumbprint <base64-encoded-certificate-thumbprint> usually comes in a hexadecimal representatation, e.g. 49AF595AE9C628BCAF5AB9D36B4EEC97DFF24458

Sites like Base64.guru or this CyberChef recipe can be used to get the correct base 64 encoded representation of the certificate thumbprint: Sa9ZWunGKLyvWrnTa07sl9/yRFg=

Everything put together

$ jwtgen -a RS256 \
         -p key.pem \
         -e 3600 \
         -c "iss=791ff0ac-1f93-47fc-8629-6b20f360036f" \
         -c "sub=791ff0ac-1f93-47fc-8629-6b20f360036f" \
         -c "aud=https://login.microsoftonline.com/0a7ab993-afa3-4a29-9b0b-f64583a4d888/oauth2/token"  \
         --headers "{\"typ\":\"JWT\",\"alg\":\"RS256\",\"x5t\":\"Sa9ZWunGKLyvWrnTa07sl9/yRFg=\"}" \
         -v

The -v commandline option produces verbose output including the encoded claims, headers and the token:

algorithm: RS256

claims:
{
  "iat": 1678549167,
  "exp": 1678552767,
  "iss": "791ff0ac-1f93-47fc-8629-6b20f360036f",
  "sub": "791ff0ac-1f93-47fc-8629-6b20f360036f",
  "aud": "https://login.microsoftonline.com/0a7ab993-afa3-4a29-9b0b-f64583a4d888/oauth2/token"
}

headers:
{
  "typ": "JWT",
  "alg": "RS256",
  "x5t": "Sa9ZWunGKLyvWrnTa07sl9/yRFg="
}

token:
eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1.... many more characters

Using the client assertion - shown in the token field - we can now try to request a token from the aouth2 token endpoint:

POST https://login.microsoftonline.com:443/0a7ab993-afa3-4a29-9b0b-f64583a4d888/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id=791ff0ac-1f93-47fc-8629-6b20f360036f
&grant_type=client_credentials
&scope=https://loitzl.sharepoint.com/.default
&client_assertion=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1......
&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer

Et voilà, we (hopefully) got a valid access_token in the response:

{
  "token_type": "Bearer",
  "expires_in": 3599,
  "ext_expires_in": 3599,
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1d...."
}

Lets try this by…

Getting all Lists from a SharePoint Site using the REST API

Using the token requested using a client assertion we can now try to get all lists from my SharePoint root site:

GET https://loitzl.sharepoint.com/_api/web/lists
Accept: application/json;odata=verbose
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOi....

The result: the first couple of fields of the /lists request.

{
  "d": {
    "results": [
      {
        "__metadata": {
          "id": "https://loitzl.sharepoint.com/_api/Web/SiteUserInfoList",
          "uri": "https://loitzl.sharepoint.com/_api/Web/SiteUserInfoList",
          "etag": "\"82\"",
          "type": "SP.List"
        },
        "FirstUniqueAncestorSecurableObject": {
[...]

✅ Yess!

Using a client secret

My IT department asked me if it’s possible to use a client secret instead, so I tried this as well.

The token endpoint request:

POST https://login.microsoftonline.com:443/0a7ab993-afa3-4a29-9b0b-f64583a4d888/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id=791ff0ac-1f93-47fc-8629-6b20f360036f
&grant_type=client_credentials
&scope=https://loitzl.sharepoint.com/.default
&client_secret=tdB8Q....

returns an access token as well:

{
  "token_type": "Bearer",
  "expires_in": 3599,
  "ext_expires_in": 3599,
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJS...."
}

If we use that token on the SharePoint REST API…

GET https://loitzl.sharepoint.com/_api/web/lists
Accept: application/json;odata=verbose
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSU...

…it replies with an “401 Unauthorized”:

Unsupported app only token.

Structure of the Bearer Token

The bearer token acquired with a client assertion contains an appidacr claim with a value of 2, the client secret acquired token has an appidacr value of 1.

I reckon that SharePoint uses this value to distinguish if the endpoint accepts the token or not.

Conclusion

In my opinion signed client assertions are a more secure method for getting bearer tokens than using client secrets (regardless of SharePoint).

If someone gets hold of the client secret in transport, the window of compromise is as long as the validity duration of the client secret (thus keep them as short as possible).

But if we lose a client assertion in transport, its only valid for the requested duration, probably just a couple of minutes or even seconds.

Nevertheless, we need to handle the certificate, which is another topic.

💙

References

Azure AD Token Generation using a Certificate Secret Client Credentials Flow. Philip Harding
Doing the same thing in .Net Core. Waldek Mastykarz
Photo by Pavel Danilyuk
JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants

Getting a Term using a Custom Property Value with PnP Core

2022-11-08T11:11:11.000Z

Objective

A nice feature of SharePoint Managed Metadata is the possibility to store custom key-value pairs as “Shared custom properties” on terms.

Using the PnP Core SDK they are also easy to add programatically:

// Add a term set
var termSet = await group
    .Sets
    .AddAsync("PnPSet1", "Set description");

// Add a term
var newTerm = await termSet
    .Terms
    .AddAsync("T1", "Description in English");

// Add a custom property with key and value
newTerm.AddProperty(
    "property1", 
    "value1"
);

// Commit changes
await newTerm.UpdateAsync();

I used this feature a lot to create hierarchical structures, e.g. for tree like site collection navigation. Each site collection url is stored as a custom property on the corresponding term.

Simple as that.

The concept is so generic that it opens up a lot of other possibilities, e.g. the key-value pairs do not have to be unique, so it can also be used for categories or classification, etc.

Retrieving Terms using custom properties

A new feature of the PnP Core SDK comes with PnP Core SDK v1.8:

Add ITermSet.GetTermsByCustomProperty method to retrieve a list of terms with a certain custom property key and value

The new method ITermSet.GetTermsByCustomProperty( key, value ) gets a list of all ITerms containing a certain custom property key and value.

// load the terms by custom property
var IList<ITerm> = await termSet
    .GetTermsByCustomProperty(
        "property1", 
        "value1"
     );

There is a section in Working with taxonomy: advanced concepts giving a little bit of context around that topic.

Slick 💪

Photo in Metadata by Ave Calvar Martinez

Retrieving Azure Application Insights using Azure CLI

2022-04-12T23:40:32.169Z

Objective

Besides the Azure Insights user interface in the Azure Portal before there is also an Azure CLI extension for retrieving and querying Azure Insights data.

This article describes a way how to retrieve that data in a background job or daemon application.

Authentication Options

There are several ways to authenticate the Azure CLI using az login.

If the login request is from a computer with a browser, a login window will automatically pop-up.

When the cli is running in a container or remote ssh session Azure CLI will automatically fall back to device authentication:

$ docker run -v ${PWD}:/workspace -it mcr.microsoft.com/azure-cli

[Container starts]

bash-5.1# az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AABBCCDD5 to authenticate.

[Sign In in Browser]

[
  {
    "cloudName": "AzureCloud",
    [..]
    "name": "Visual Studio Ultimate mit MSDN",
    "state": "Enabled",
    "tenantId": "0a7ab993-afa3-4a29-9b0b-f64583a4d888",
    [..]
  }
]

bash-5.1#

For automation purposes it can be necessary to authenticate within azure cli using a client secret…

$ az login \
  --service-principal -u c4fe6c64-4e6a-4997-9a4f-7d96b30dfb51 \ # App Id
  -p iNr....GaaNE \ # Client Secret
  --tenant 0a7ab993-afa3-4a29-9b0b-f64583a4d888 # Tenant id or name

…or with a client certificate in pem format:

$ az login \
  --service-principal -u c4fe6c64-4e6a-4997-9a4f-7d96b30dfb51 \ # App Id
  -p ClientCert.pem \ # Client Certificate
  --tenant 0a7ab993-afa3-4a29-9b0b-f64583a4d888 # Tenant id or name

Information on how to obtain a certificate in pem format using openssl can be found here

Find the Azure Insights Resource

Now we can try to find the correct azure insights resource.

With az group list resource groups where the service principal has access can be listed.

$ az group list 
{ [..] },
{
    "id": "/subscriptions/4664bbed-abef-4e8b-bd4a-462f757c712c/resourceGroups/rg-dev",
    "location": "switzerlandnorth",
    "managedBy": null,
    "name": "dev",
    "properties": {
      "provisioningState": "Succeeded"
    },
    "tags": null,
    "type": "Microsoft.Resources/resourceGroups"
}
{ [..] },
{ [..] }

When the resource group name is known we can list the contents of a resource group using az resource list

$ az resource list \
  --resource-group rg-dev \ # the resource group from the previous step
  --resource-type Microsoft.Insights/components # azure insights type

[
  {
    "changedTime": "2022-02-24T10:17:15.282736+00:00",
    "createdTime": "2022-02-24T10:07:14.668422+00:00",
    "extendedLocation": null,
    "id": "/subscriptions/4664bbed-abef-4e8b-bd4a-462f757c712c/resourceGroups/rg-dev/providers/Microsoft.Insights/components/dev",
    "identity": null,
    "kind": "web",
    "location": "switzerlandnorth",
    "managedBy": null,
    "name": "dev",
    "plan": null,
    "properties": null,
    "provisioningState": "Succeeded",
    "resourceGroup": "rg-dev",
    "sku": null,
    "tags": {},
    "type": "Microsoft.Insights/components"
  }
]

The value of the id can be used to retrieve monitoring events.

Retrieving Events

Using az monitor app-insights events we can now retrieve events from Azure Insights by passing the id retrieved in the previous step as the --app parameter.

$ az monitor app-insights events show \
  --app /subscriptions/4664bbed-abef-4e8b-bd4a-462f757c712c/resourceGroups/rg-dev/providers/Microsoft.Insights/components/dev \
  --type traces \
  --offset 24h

Setting the --offset parameter will retrieve application traces within the last 24 hours.

{[..]},
{
      "ai": {
        "appName": "dev",
        "sdkVersion": "azurefunctions: 3.5.2.0",
        [..]
      },
      "application": {[..]},
      "client": {
        [..],
        "type": "PC"
      },
      "cloud": {
        "roleInstance": "...",
        "roleName": "dev"
      },
      "count": 1,
      "customDimensions": { [..] },
      "customMeasurements": null,
      "id": "dbf73851-ba43-11ec-a27c-af008d3d954f",
      "operation": {
        "name": "PeopleSyncJob",
        "parentId": "2bd1380092e0fd47",
        [..]
      },
      "session": {
        "id": ""
      },
      "timestamp": "2022-04-12T09:35:02.767000+00:00",
      "trace": {
        "message": "🍰 i:0#.f|membership|gradya@loitzl1.onmicrosoft.com's Birthday: '3/8/2000 12:00:00 AM'",
        "severityLevel": 0
      },
      "type": "trace",
      "user": {[..]}
    }
{[..]},
{[..]}

The field trace.message contains the custom log entry we might be interested in.

Executing Analytics Queries

It is also possible to run kusto queries within Azure CLI using az monitor app-insights query

$ az monitor app-insights query \
  --app fc162434-7ef9-4d1c-a9d7-3ac820c38800 \
  --offset 24h \
  --analytics-query 'traces | search "*grady*"'

This returns a json table structure

{
  "tables": [
    {
      "columns": [..],
      "name": "PrimaryResult",
      "rows": [..]
    }
  ]
}

where columns contains a description of the row’s data,

{
 "columns": [
   {
     "name": "$table",
     "type": "string"
   },
   {
     "name": "timestamp",
     "type": "datetime"
   },
   {
     "name": "message",
     "type": "string"
   },
   [...]

and rows contains the actual data:

 [
   "traces",
   "2022-04-11T16:00:02.770299Z",
   "🍰 i:0#.f|membership|gradya@loitzl1.onmicrosoft.com's Birthday: '3/8/2000 12:00:00 AM'",
   [..]

Correlate Log entries

If we have something that can be used to correlate log entries over one request or operation, a full trace can be retrieved like this:

$ az monitor app-insights query \
  --app fc162434-7ef9-4d1c-a9d7-3ac820c38800 \
  --offset 24h \
  --analytics-query 'traces | where customDimensions.prop__SyncPropertiesId == "76b036f3-3083-4472-8958-171c46fb8e02"'

together with the magic of jq we can transform the result into a complete trace of a certain operation:

 $ ... | jq '.tables[0] | .rows | .[][1]'

"Syncing user 'GradyA@loitzl1.onmicrosoft.com'"
"Creating ACSAppOnlyContext for 'https://loitzl1-admin.sharepoint.com' and 'eca24dc1-1e18-4ed0-94b7-f9c269caf540'"
"✔ Successfully resolved 'i:0#.f|membership|gradya@loitzl1.onmicrosoft.com'"
"Users Country 'Switzerland'"
"🍰 i:0#.f|membership|gradya@loitzl1.onmicrosoft.com's Birthday: '3/8/2000 12:00:00 AM'"
"✔ Successfully synced user 'i:0#.f|membership|gradya@loitzl1.onmicrosoft.com'"

Outlook

Being able to retrieve and analyze Azure Insights data as a daemon app brings interesting opportunities for log analytics and alerting.

It could also be used for higher level integration testing in projects with a low level of testability.

Syncing MS Graph Properties with SharePoint User Profile from Azure Function

2022-03-16T23:40:32.169Z

Azure Active Directory SharePoint User Profile

🍓

Introduction

In SharePoint Online only syncs a predefined set of properties from an Azure Active Directory User object to the SharePoint UserProfile and as mentioned in the FAQ section no additional properties can be added.

But it is still possible to add new User Profile Properties in SharePoint Online.

The newly created property becomes a Crawled Property and can be mapped to a Managed Property and surfaced in the SharePoint Online Search Schema.

Using a timer-triggered Azure Function it is possible to synchronize Azure Active Directory properties of a user object (or anything else) with the SharePoint Online User Profile.

This example shows how to get the ‘Country or Region’ property of the AAD User object synced with a SharePoint User Profile custom property and a Managed Property in SharePoint Online Search.

Creating the User Profile Property

Navigate to SharePoint Admin Center -> More Features -> User Profiles

https://<tenant>-admin.sharepoint.com/
      _layouts/15/TenantProfileAdmin/
           ManageUserProfileServiceApplication.aspx

Click “Manage User Properties”
Add a “New Property”
Give it a name
Be sure that the “Default Privacy Setting” is set to “Everyone”

Be sure that in “Search Settings” the checkbox “Indexed” is set

Creating the Search Managed Property

Navigate to SharePoint Admin Center -> More Features -> Search
Click “Manage Search Schema”
Select “Managed Properties”

https://<tenant>-admin.sharepoint.com/
       _layouts/15/searchadmin/
           ta_listmanagedproperties.aspx?level=tenant

Look for an existing “RefineableStringXYZ” (I used “RefinableString120” in this example)
Add an Alias, e.g., CountryOrRegion

Click “Add a Mapping” to map the Crawled Property “People:CustomCountry” on the “Managed Property”

Note: It may take several hours for the Crawled Property “People:CustomCountry” to show up.

Setup Authentication for the Azure Function

This topic is addressed in another article Creating SharePoint Terms using the App Context with the new PnP Core SDK.

A PowerShell Script can be found in the Github Repository.

The Azure Function

The Azure Function implementation that synchronizes the “Country or Region” property can be found in this GitHub Repository

Utility.ResolvePrincipal can be used to receive the login of the user

var resolvedPrincipal =
  Utility.ResolvePrincipal(
      clientContext,
      clientContext.Web,
      user.Mail,
      PrincipalType.User,
      PrincipalSource.All,
      null,
      true); // true - as input is only the email

await clientContext.ExecuteQueryRetryAsync();
PrincipalInfo principal = resolvedPrincipal.Value;

string loginName = principal.LoginName;

With the loginName we can then update the CustomCountry property.

  var peopleManager = new PeopleManager(clientContext);
  peopleManager.SetSingleValueProfileProperty(
      loginName,
      "CustomCountry",  // The custom UserProfile property
      user.Country);
  clientContext.Load(peopleManager);
  await clientContext.ExecuteQueryRetryAsync();

The Problem

PeopleManager can be used to manage SharePoint Online UserProfile Properties.

The Azure AD app registration that is used for accessing the graph cannot be used to access the SharePoint User Profile.

Since we are authenticating the Azure Function using the PnP Core SDK we can use PnP.Framework.AuthenticationManager.GetACSAppOnlyContext to get a ClientContext.

  public ClientContext GetACSAppOnlyContext(
    string siteUrl, string appId, string appSecret)

The appId and appSecret required for this call are from a SharePoint Add-In Registration.

So, a “proxy” SharePoint Add-In is required for that:

Let’s first create a new Add-In using AppRegNew.aspx:

    https://<tenant>.sharepoint.com
          /_layouts/15/appregnew.aspx

Generate Client Id - note the id for later use
Generate Client Secret - note the secret for later use
Title: ‘Graph SyncJob Delegate App’
App Domain: “www.localhost.com”
Redirect URL: “https://www.localhost.com”

Then the permissions need to be assigned using:

  https://<tenant>-admin.sharepoint.com
           /_layouts/15/appinv.aspx

Enter the App Id from the previous step and click on ‘Lookup’.
In the Add-Ins’s permissions request, enter the followings:

<AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/social/tenant" Right="FullControl" />
</AppPermissionRequests>

Click on “Create”
Click on “Trust”

This will request App Only Permissions for writing to the SharePoint User Profile.

Enabling App Only Authentication

App-Only authentication is deactivated by default, but it can be enabled with this PnP PowerShell command:

    Connect-PnPOnline -Url https://<tenant>-admin.sharepoint.com -DeviceLogin
    Set-PnPTenant -DisableCustomAppAuthentication $false

Results

After the job ran successfully the UserProfile contains a value from the Azure AD field “Country or Region” on the SharePoint UserProfile…

SharePoint User Profile

… and in Search, we have the RefinableString120 property with the same value.

(Submit-PnPSearchQuery `
  -SourceId b09a7990-05ea-4af9-81ef-edfab16c4e31       # User Profiles              `
  -Query "DocId:17592245537205"                        # My Account in this Tenant  `
  -SelectProperties RefinableString120,CountryOrRegion # mapped to CustomCountry    `
).ResultRows

Key                      Value
---                      -----
RefinableString120       Erdbeerland
CountryOrRegion          Erdbeerland
DocId                    17592245537205
GeoLocationSource        EUR
contentclass             urn:content-class:SPSPeople
AccountName              i:0#.f|membership|martin@loitzl.onmicrosoft.com
Department               Development
JobTitle                 Developer
LastModifiedTime         03/17/2022 09:54:21
Path                     https://loitzl-my.sharepoint.com/Person.aspx?accountname=i:0%23.f|membership|martin@loitzl.onmicrosoft.com
PictureURL               https://loitzl-my.sharepoint.com/User Photos/Profile Pictures/martin_loitzl_onmicrosoft_com_MThumb.jpg
PreferredName            Martin Loitzl
SipAddress               martin@loitzl.onmicrosoft.com
WorkEmail                martin@loitzl.com
Title                    Martin Loitzl

Final Notes

The function uses PnP.Core Authentication to connect to MS Graph and a SharePoint “Proxy” Add-In with AppOnly Permissions to write to the SharePoint User Profile Service:

And Kudos to Darius for figuring that out in one of our customer projects 👍🦄

Troubleshooting SharePoint Online Search

2021-11-27T23:40:32.169Z

Introduction

Lately I had the chance to troubleshoot an issue we had with SharePoint Online Search. This article summarizes what I have learned with a small example scenario.

First lets add some site columns

one simple Text column SampleTextProperty1
and a managed metadata column SampleTaxonomyProperty1 to make it more intesting

Then add the site columns to a document library:

Upload documents and set some metadata:

Analyze Crawl Log

We can immediately check if the documents get crawled with the Get-PnPSearchCrawlLog PnP Powershell Commandlet, the -Filter parameter allows to filter by site url for example.

$ Connect-PnPOnline -Url https://loitzl2.sharepoint.com/sites/ManagedProperties/ -DeviceLogin
[...]

$ Get-PnPSearchCrawlLog `
   -Filter https://loitzl2.sharepoint.com/sites/ManagedProperties

[...]

Url             : https://loitzl2.sharepoint.com/sites/ManagedProperties/Shared Documents/Test Page.pdf
CrawlTime       : 10/12/2021 6:22:25 AM
ItemTime        : 10/12/2021 1:22:10 PM
LogLevel        : Success
Status          :
ItemId          : 58807014
ContentSourceId : 1

Url             : https://loitzl2.sharepoint.com/sites/ManagedProperties/Shared Documents/Invoice.docx
CrawlTime       : 10/12/2021 6:22:24 AM
ItemTime        : 10/12/2021 1:21:54 PM
LogLevel        : Success
Status          :
ItemId          : 58807017
ContentSourceId : 1

[..]

✅ Seems to be OK

The permissions for reading the search crawl log can be set in the SharePoint Admin Center:

/_layouts/15/searchadmin/crawllogreadpermission.aspx

Re-crawl a Document Library

In case we need to tell the search crawler to crawl the document library again the Request-PnPReIndexList pnp powershell commandlet can be used:

$ Request-PnPReIndexList -Identity "Documents"

The same function is surfaced in the UI:

https://docs.microsoft.com/en-us/sharepoint/crawl-site-content

🤓 What happens behind the scenes ist, that the vti_searchversion property bag value gets increased:

const string reIndexKey = "vti_searchversion";
var searchversion = 0;

if (list.PropertyBagContainsKey(reIndexKey))
{
  searchversion = (int)list.GetPropertyBagValueInt(reIndexKey, 0);
}
try
{
  list.SetPropertyBagValue(reIndexKey, searchversion + 1);
}
[...]

see Microsoft.SharePoint.Client.ListExtensions

And, btw, the same works for whole sites with the Request-PnPReIndexWeb commandlet which writes vti_searchversion to the SP.Web property bag.

see Microsoft.SharePoint.Client.WebExtensions

Managed Properties

After giving the search crawler some time, two new managed properties will probably show up:

Using the Submit-PnPSearchQuery search results can be retrieved:

$ Submit-PnPSearchQuery ` 
     -Query "SampleTextProperty1OWSTEXT:'An Invoice'" 

GroupTemplateId              : 
ItemTemplateId               : 
Properties                   : {[RenderTemplateId, ~sitecollection/_catalogs/masterpage/Display Templates/Sea
                               rch/Group_Default.js], [QueryModification, SampleTextProperty1OWSTEXT:'An Invo
                               ice' -ContentClass=urn:content-class:SPSPeople], [ExecutionTimeMs, 0], [StartR
                               ecord, 0]…}
QueryId                      : 0fc5abbe-875a-2e7c-32dc-e49b00dcfb21
QueryRuleId                  : 00000000-0000-0000-0000-000000000000
ResultRows                   : {System.Collections.Generic.Dictionary`2[System.String,System.Object]}
ResultTitle                  : 
ResultTitleUrl               : 
RowCount                     : 1
TableType                    : RelevantResults
TotalRows                    : 1
TotalRowsIncludingDuplicates : 1
TypeId                       : {6780df59-1036-4912-829b-432354f22656}

We receive exactly the one item in the document library.

$ (Submit-PnPSearchQuery `
  -Query "SampleTextProperty1OWSTEXT:'An Invoice'") `
     .ResultRows `
       .OriginalPath

https://.../ManagedProperties/Shared Documents/Invoice.docx

We can also retrieve the value of the Managed Property SampleTextProperty1OWSTEXT:

$ (Submit-PnPSearchQuery `
  -Query "SampleTextProperty1OWSTEXT:'A printer test page'" `
  -SelectProperties SampleTextProperty1OWSTEXT) `
     .ResultRows `
        .SampleTextProperty1OWSTEXT

A printer test page

One problem that arises here that the other managed property owstaxIdSampleTaxonomyProperty11 contains the ids of the taxonomy tree.

$ (Submit-PnPSearchQuery `
     -Query "SampleTextProperty1OWSTEXT:'A printer test page'" `
     -SelectProperties owstaxIdSampleTaxonomyProperty11) `
        .ResultRows `
           .owstaxIdSampleTaxonomyProperty11        

GP0|#318e3a7f-0f62-4d57-bf0e-e8b3d693ba6e;L0|#0318e3a7f-0f62-4d57-bf0e-e8b3d693ba6e|R＆D;GTSet|#8ed8c9ea-7052-4c1d-a4d7-b9c10bffea6f

Search also creates a crawled property containing the label of a taxonomy field:

So lets create a new managed property named SampleTaxonomyProperty1 and map the crawled property ows_SampleTaxonomyProperty11 to it:

And retrieve it:

$ (Submit-PnPSearchQuery ``
   -Query "SampleTextProperty1OWSTEXT:'A printer test page'" `
   -SelectProperties SampleTaxonomyProperty1) `
     .ResultRows `
       .SampleTaxonomyProperty1

R& D

Thats a lot more readable ✌️

Note: Might take a couple of minutes after mapping for the value to show up.

Search Query Tool

Another nice tool is the SharePoint Search Query Tool which has been archived in the PnP github repo. It allows doing the same things as with Submit-PnPSearchQuery, especially getting all managed properties can be helpful:

There is also a possibility to retrieve (at least the names of) all crawled properties of the item:

Conclusion

Working with Search in SharePoint Online with a SharePoint on-prem background feels odd at the beginning since its easy to just access the ULS Logs.

Actually there are a plenty of possbilities around to analyze Managed Properties, Crawled Properties and the Crawl Log itself.

I will update that article if I find some new tricks.

💙

Creating SharePoint Terms using the App Context with the new PnP Core SDK

2021-05-23T23:40:32.169Z

Introduction

Lately I had the chance to work with the new PnP Core SDK.

Its running under dot net core, which is an awesome improvement compared to the older PnP Sites Core which I used a lot in the past couple of years ❤️

The methods for accessing the termstore leverage the new taxonomy endpoint of MS Graph beta endpoint.

The article shows

the setup of an Azure AD App using the PnP Powershell Module and
an Azure Function accessing the Term Store using client certificate authentication.

Setup Authentication

Setting up the Azure AD App for this is straight forward using the Register-PnPAzureADApp PowerShell commandlet.

Im am doing this with macOS Big Sur, so first we have to install PowerShell:

$ brew install --cask powershell

That seemed to work, no lets start PowerShell.

$ pwsh

Install the PnP PowerShell Module

Install-Module -Name "PnP.PowerShell"

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from 'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): y

The App can be created and trusted in Azure Active Directory using the Register-PnPAzureADApp PnP commandlet. This commandlet can also create a self-signed certificate out of the box, but this only works on Windows machines.

On Linux/Mac based system we can provide our own certificate using the following openssl commands:

# Create the certificate
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365  
# Convert to pfx file
$ openssl pkcs12 -export -out certificate.pfx -inkey key.pem -in cert.pem

The command in line 4 converts the certificate created in line 2 to a pfx file, which should have a password.

Now the Client Application can be registered using the Register-PnPAzureADApp command. The password used for creating the pfx file has now to be passed as the CertificatePassword (line 5) parameter.

The -Scopes parameter in line 8 is requesting the necessary permissions.

Register-PnPAzureADApp `
  -ApplicationName TestApp1 `
  -Tenant <tenant>.onmicrosoft.com ` 
  -CertificatePath certificate.pfx `
  -CertificatePassword (ConvertTo-SecureString -String "<cert password>" -AsPlainText -Force) `
  -Username "<user>@<tenant>.onmicrosoft.com" `
  -Password (ConvertTo-SecureString -String "<password>" -AsPlainText -Force) `
  -Scopes "SPO.TermStore.ReadWrite.All", "MSGraph.TermStore.ReadWrite.All"

On Windows systems the consent window will open automatically. On Linux/Mac based systems the consent Url is written to STDOUT and should be pasted into a browser window.

https://login.microsoftonline.com
      /loitzl.onmicrosoft.com/v2.0
        /adminconsent
    ?client_id=791ff0ac-1f93-47fc-8629-6b20f360036f
    &scope=https://graph.microsoft.com/.default
    &redirect_uri=http://localhost

The return value contains the necessary values for authentication:

The certificate.pfx file has been renamed to <AppName>.pfx and <AppName>.cer (here: TestApp1)
The AzureAppId/ClientId of the newly created application
The Certificate Thumbprint
And the Base64 Encoded certificate data

This data is needed for configuring the Azure Function authentication.

The Function

After creating a new function project and adding the PnP Core SDK package with

$ dotnet add package PnP.Core --version 1.1.0

adjust the local.settings.json file with the values of the result from Register-PnPAzureADApp result above:

SiteUrl: Url to connect to
TenantId: can be retrieved with Get-PnPTenantId
ClientId: the AzureAppId/ClientId from Register-PnPAzureADApp
CertificateThumbPrint: also from Register-PnPAzureADApp

In order to use the PnP Core SDK using dependency injection create a Startup class inheriting from FunctionsStartup as described in the MSDN Docs.

The full source code can be found on my Github.

public class Startup : FunctionsStartup
{
    public override void Configure(IFunctionsHostBuilder builder)
    {
        var config = builder.GetContext().Configuration;
        var settings = new Settings();
        config.Bind(settings);
        builder.Services.AddPnPCore(options =>
        {
            options.DisableTelemetry = true;
            var authProvider = new X509CertificateAuthenticationProvider(
                settings.ClientId.ToString(),
                settings.TenantId.ToString(),
                StoreName.My,
                StoreLocation.CurrentUser,
                settings.CertificateThumbprint);
            options.DefaultAuthenticationProvider = authProvider;
            options.Sites.Add("Default",
                   new PnPCoreSiteOptions
                   {
                       SiteUrl = settings.SiteUrl.ToString(),
                       AuthenticationProvider = authProvider
                   });
        });
    }
}

In lines 10-17 client certificate authentication gets configured, a Default connection is added with the SiteUrlconfigured in the local.settings.json in line 18 to 23.

This allows us to use constructor injection to retrieve a IPnPContextFactory.

public class CreateTerm
{

  private readonly IPnPContextFactory _pnpContextFactory;
  public CreateTerm(IPnPContextFactory pnpContextFactory)
  {
    _pnpContextFactory = pnpContextFactory;
  }
  ...

The IPnPContextFactory can now be used to create a PnPContext instance, retrieve the first ITermGroup and first ITermSet for simplicity (use the *ById() methods in real life).

In the ITermSet we then create a new Term with termSet.Terms.AddAsync(name) method (line 8).

using (var pnpContext = await _pnpContextFactory.CreateAsync("Default"))
{
    _logger.LogInformation("Creating Term '{DefaultLabel}'", name);
    try
    {
        var termGroup = pnpContext.TermStore.Groups.First();
        var termSet = termGroup.Sets.First();
        var newTerm = await termSet.Terms.AddAsync(name);
        return new OkObjectResult(new { id = newTerm.Id });
    }
    catch (MicrosoftGraphServiceException ex)
    {
        _logger.LogError((Exception)ex, ex.Error.ToString());
        return new BadRequestObjectResult(ex.Error);
    }
}

After starting the function in debug mode with F5, we see, that this does not work actually.

We get a MicrosoftGraphServiceException with the message

❌ “The current user has insufficient permissions to perform this operation.”

HttpResponseCode: 403
Code: accessDenied
Message: The current user has insufficient permissions to perform this operation.
ClientRequestId: a07d1a32-410f-4397-8157-e329310c7fb5
date: 2021-05-22T08:55:35
client-request-id: a07d1a32-410f-4397-8157-e329310c7fb5

👉 PnP.Core: Microsoft Graph service exception.

The Problem

Accessing the SharePoint Termstore in the context of the Application (app-only context) needs an old trick that is still possible:

Add the app principal “app@sharepoint” as Term Group Manager, and it works.

A google search with “i:0i.t|00000003-0000-0ff1-ce00-000000000000|app@sharepoint” reveals, that there are sometimes problems with this approach but that it seems that it is widely used.

Good troubleshoting tips can be found in the Github issue’s discussion.

Conclusion

By the way, all this was created on a new Apple MacBook Pro M1. I really like the fact that more and more of the M365 developer and engineering workloads are now available cross-platform.

Configuration the PnP Core SDK using the

AddPnPCore(
  this IServiceCollection services, 
  Action<PnPCoreOptions> options)

extension method and the integration into the .net core dependency injection world is also a great feature.

Setup mini Kubernetes Rancher K3S on Raspberry OS Lite

2021-02-05T22:40:32.169Z

After finishing the hardware part of the cluster, let’s see how to setup Kubernetes.

Versions

Raspberry Pi OS Lite 2020-08-20 Release Notes

Rancher K3S 1.20.0 Product Page

Create the SD Cards

List the partitions in MacOS Terminal using diskutil list.

$ diskutil list

Be really really (really!) sure, that you take the right volume. It also helps if its named properly in Finder.

In this case it has the number 2.

Lets unmount it:

$ diskutil unmountDisk /dev/disk2

Now lets go to the danger zone. dd the downloaded img file to the SD Card device.

Be sure that you get the right volume number /dev/rdisk->2<-. Using /dev/rdisk instead /dev/disk just makes the process a little faster.

$ sudo dd if=2020-08-20-raspios-buster-arm64-lite.img of=/dev/rdisk2 bs=8m

The output of dd is quite reduced, can make one nervous…

… on MacOS hit Ctrl+T to get some information about the progress.
… on most Linux distros there should be a command line option status=progress.

$ dd if=/dev/urandom of=/dev/null count=131072 bs=1024 status=progress
129695744 bytes (130 MB, 124 MiB) copied, 4 s, 32.4 MB/s # progress
131072+0 records in  # records read
131072+0 records out # records written
134217728 bytes (134 MB, 128 MiB) copied, 4.13999 s, 32.4 MB/s  # totals, duration and speed

_{https://askubuntu.com/questions/215505/how-do-you-monitor-the-progress-of-dd} _{https://www.raspberrypi.org/documentation/installation/installing-images/mac.md}

Before pulling the SD Card out of the slot, we can already set some necessary boot options for running Kubernetes:

enable cgroups

# Assuming that the new card has a boot partition that is mounted under /Volumes/boot
$ echo -n "cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1" >> /Volumes/boot/cmdline.txt

And let’s enable ssh since we are in a headless mode:

# Assuming that the new card has a boot partition that is mounted under /Volumes/boot
$ touch /Volumes/boot/ssh

First boot

Put the cards into the Pis and start them up with the Ethernet cables connected.

Find a way to determine the IP addresses, for example by logging into your router Web UI. Or if you like the more Linux way using arp commands:

Immediately change the password!

The default password currently is “raspberry” - by the way.

$ ssh pi@192.168.1.x
$ passwd

Update the Pis

$ sudo rpi-update
$ sudo reboot

K3S first contact

Now we can check if the host OS is any good for running Rancher K3S.

$ curl -sfL https://raw.githubusercontent.com/rancher/k3s/master/contrib/util/check-config.sh | sh -

In my case it complained about, that iptables is too new. Raspberry OS has a simple way to fix this:

$ update-alternatives --set iptables /usr/sbin/iptables-legacy

Install K3S

The actual setup is unbelievably easy, just run:

Master Node

$ sudo curl -sfL https://get.k3s.io | sh -

Worker Nodes

Get the node token on the master node

$ sudo cat /var/lib/rancher/k3s/server/node-token

Run this command with the ip of the master node and the node-token from above command.

$ curl -sfL http://get.k3s.io | K3S_URL=https://<MASTER NODE IP>:6443 \
                      K3S_TOKEN=<NODE TOKEN> sh -

Done!

Don’t believe it?

$ sudo kubectl get nodes
NAME                 STATUS   ROLES                  AGE    VERSION
barbarachristensen   Ready    control-plane,master   4m4s   v1.20.0+k3s2
janerose             Ready    <none>                 4m4s   v1.20.0+k3s2
maryireland          Ready    <none>                 4m4s   v1.20.0+k3s2
dianabeverley        Ready    <none>                 4m4s   v1.20.0+k3s2

😎

Convenience Features

Non-root access

Enable non-root access to kubectl:

$ cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config

Bash auto completion

Enable bash auto completion:

_{The link contains also information for other OSes.}

$ sudo apt-get install bash-completion
$ echo 'source <(kubectl completion bash)' >>~/.bashrc
$ echo 'alias k=kubectl' >>~/.bashrc
$ echo 'complete -F __start_kubectl k' >>~/.bashrc

Shutdown remotely

#!/bin/sh
for i in `cat nodes.lst`;
	do /usr/bin/ssh -t $i /usr/bin/sudo /sbin/poweroff;
done

$ cat nodes.lst
192.168.1.60
192.168.1.61
192.168.1.62
192.168.1.63

To avoid entering a password for each node, copy the ssh key to each node:

$ for i in `cat nodes.lst`;do /usr/bin/ssh-copy-id $i;done

_{from: https://www.simplylinuxfaq.com/2015/02/how-to-shutdown-remote-system-via-ssh-without-password.html}

After getting passwordless authentication to work it might be a good idea to password authentication completely.

Portable Kubernetes Cluster based on Raspberry Pi 4 and Rancher K3S
Setup mini Kubernetes Rancher K3S on Raspberry OS Lite (this)
Storage class and nfs provisioner
Setting up a four times redundant gluster volume with mini USB sticks
Automatically provision Gluster volumes with Heketi

Bringing my hacked weblog back to life

2021-02-01T22:40:32.169Z

You’ve been busted!

Due to poorly developed plugins, Wordpress blogs are prone to hacking.

Luckily I wasn’t using any of those most dangerous Wordpress plugins mentioned in this article, but just like many others, my weblog got abused for cross-site-request-forgeries and phishing - using php files that have been uploaded years before through exploits.

At the end my account got locked and I lost interest a little.

But, I still had some old backups.

On my beloved homegrown k8s cluster I was able to fire up some old php containers, a new mariadb and phpmyadmin.

_CloudSkew

I restored the backup files into the persistent volume, imported the SQL dumps to mariadb, some more tweaks and we are ready to go.

$ kustomize edit set namespace old-blog
$ kustomize build | kubectl apply -f -

$ kubectl run -i --rm --image=arm64v8/mariadb:10.5 --restart=Never mariadb-client -- mysql -h mariadb -ppassword -e "create database wordpressdb"
$ kubectl run -i --rm --image=arm64v8/mariadb:10.5 --restart=Never mariadb-client -- mysql -h mariadb -ppassword wordpressdb < wordpressdb.sql

$ kubectl run -i --rm --image=arm64v8/mariadb:10.5 --restart=Never mariadb-client -- mysql -h mariadb -ppassword -e "create database gallery2db"
$ kubectl run -i --rm --image=arm64v8/mariadb:10.5 --restart=Never mariadb-client -- mysql -h mariadb -ppassword gallery2db < loitzlco-gallery2.sql

Comes in quite quite handy for (manually) migrating and updating (dead links, etc.) to my new statically generated Lumen.

Setting up a four times redundant gluster volume with mini USB sticks

2021-01-28T22:40:32.169Z

What I really really need is a super redundant storage cluster on my Pi cluster using plain USB sticks.

This might sound a little bit over-engineered, but having a Gluster with 4 USB sticks - one for each Raspberry - allows to run the Cluster with some nodes switched off to safe some energy.

The architecture is as simple as this: Gluster - Heketi Provisioner - Kubernetes Storage Class

The first step (this article) is to create the storage cluster using USB sticks.

This is pretty straight forward and completely based on other blog articles cited at the end.

Setup the Gluster Server

Install the glusterfs-server package on each node

$ sudo apt-get install glusterfs-server

Host file entries allow all nodes to find each other

$ sudo nano /etc/hosts

192.168.1.60    gluster0
192.168.1.61    gluster1
192.168.1.62    gluster2
192.168.1.63    gluster3

Check if glusterd.service has been correctly set-up …

$ sudo systemctl status glusterd.service
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/lib/systemd/system/glusterd.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:glusterd(8)

… then start and enable the service on each node.

$ sudo systemctl start glusterd.service
$ sudo systemctl enable glusterd.service

Now probe the nodes

$ sudo gluster peer probe gluster1
peer probe: success.

$ sudo gluster peer status
Number of Peers: 3

Hostname: 192.168.1.61
Uuid: e34e779c-f0f4-453b-bd9f-103654470614
State: Peer in Cluster (Connected)
Other names:
gluster1

Hostname: gluster2
Uuid: cc244b01-1884-4d2d-a213-29f283f3067a
State: Peer in Cluster (Connected)

Hostname: gluster3
Uuid: a2faf317-47bb-414b-9a87-7e2240b210cd
State: Peer in Cluster (Connected)

Prepare the USB Sticks

The USB sticks need to be formatted for Gluster. First list all block devices with lsblk and be sure to pick the right one

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    1 57.6G  0 disk
`-sda1        8:1    1 57.6G  0 part
mmcblk0     179:0    0 29.8G  0 disk
|-mmcblk0p1 179:1    0  256M  0 part /boot
`-mmcblk0p2 179:2    0 29.6G  0 part /

Run the following commands to partition the USB sticks

$ sudo fdisk -w auto /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): g
Created a new GPT disklabel (GUID: 0F840961-BAB8-064B-ABFE-C02CA2D346A0).
The old dos signature will be removed by a write command.

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-120831966, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-120831966, default 120831966):

Created a new partition 1 of type 'Linux filesystem' and of size 57.6 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

The newly created partition can then be formatted with the XFS filesystem

$ sudo mkfs.xfs -f -L glstrbrck0 /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=4, agsize=3775935 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=15103739, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=7374, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Create the mountpoint

$ sudo mkdir -p /data/glusterfs/glstrbrck/0

Determine the partition UUID and write it to the fstab so that its get mounted automatically when booting

$ printf $(sudo blkid -o export /dev/sda1|grep PARTUUID)" /data/glusterfs/glstrbrck/0 xfs defaults,noatime 1 2\n" | sudo tee -a /etc/fstab

And mount (or reboot)

$ sudo mount /data/glusterfs/glstrbrck/0

The -oflag of lsblk can be used show the details of the newly created partition:

$ lsblk -o name,size,type,fstype,mountpoint,label
NAME         SIZE TYPE FSTYPE MOUNTPOINT                  LABEL
sda         57.6G disk                                    
`-sda1      57.6G part xfs    /data/glusterfs/glstrbrck/0 glstrbrck0
mmcblk0     29.8G disk                                    
|-mmcblk0p1  256M part vfat   /boot                       boot
`-mmcblk0p2 29.6G part ext4   /                           rootfs

All right so far

Create the Gluster Volume

After performing the steps on all nodes, a new Gluster Volume can be created - with 4 replicas in my case.

$ sudo gluster volume create glustervol0 replica 4 gluster0:/data/glusterfs/glstrbrck/0/brick gluster1:/data/glusterfs/glstrbrck/0/brick gluster2:/data/glusterfs/glstrbrck/0/brick gluster3:/data/glusterfs/glstrbrck/0/brick

$ sudo gluster volume start glustervol0

Lets check the results:

List the Volumes…

$ sudo gluster volume list
glustervol0

… and display volume details

$ sudo gluster volume info glustervol0
 
Volume Name: glustervol0
Type: Replicate       # <- A replicated volume
Volume ID: 066c49bb-9c6b-48b0-8f4a-2e9c72e1aa40
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: gluster0:/data/glusterfs/glstrbrck/0/brick  # <- 4 bricks
Brick2: gluster1:/data/glusterfs/glstrbrck/0/brick
Brick3: gluster2:/data/glusterfs/glstrbrck/0/brick
Brick4: gluster3:/data/glusterfs/glstrbrck/0/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

😎 - Awesome

Some Tests

Mount the the gluster volume on one of the nodes:

$ sudo mkdir /storage-pool
$ sudo mount -t glusterfs gluster0:/glustervol0 /storage-pool

Create some files

$ cd /storage-pool/
$ sudo touch file_{0..9}.test

Mount the gluster volume on another host, and check the result.

$ sudo mount -t glusterfs gluster1:/glustervol0 /storage-pool
$ cd /storage-pool
$ ls

storage-pool $ ls -la

drwxr-xr-x  3 root root 4096 Jan 25 16:47 .
drwxr-xr-x 23 root root 4096 Jan 17 17:52 ..
-rw-r--r--  1 root root    0 Jan 17 17:53 file_0.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_1.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_2.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_3.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_4.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_5.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_6.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_7.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_8.test
-rw-r--r--  1 root root    0 Jan 17 17:53 file_9.test

That works!

Performance

Now lets check the performance…

Writing random data

$ sudo dd if=/dev/urandom of=/storage-pool/file1.rnd count=65536 bs=1024

65536+0 records in
65536+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 14.5248 s, 4.6 MB/s

… increasing the block size helps.

sudo dd if=/dev/urandom of=/storage-pool/file2.rnd count=512 bs=131072

512+0 records in
512+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 2.37264 s, 28.3 MB/s

Whats actually the read speed of urandom?

$ dd if=/dev/urandom of=/dev/null count=65536 bs=1024

65536+0 records in
65536+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 1.32923 s, 50.5 MB/s

Pretty nice for an embedded device.

Reading data

Read a file…

$ dd if=/storage-pool/file2.rnd of=/dev/null bs=131072
512+0 records in
512+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.595213 s, 113 MB/s

… and read the file a second time reveals the caching performance.

$ dd if=/storage-pool/file2.rnd of=/dev/null bs=131072
512+0 records in
512+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.0385229 s, 1.7 GB/s

Setting up a 4 times redundant storage array with Gluster went pretty straight forward due to the follwing awesome articles:

Portable Kubernetes Cluster based on Raspberry Pi 4 and Rancher K3S
Setup mini Kubernetes Rancher K3S on Raspberry OS Lite
Storage class and nfs provisioner
Setting up a four times redundant gluster volume with mini USB sticks (this)
Automatically provision Gluster volumes with Heketi

Portable Kubernetes Cluster based on Raspberry Pi 4 and Rancher K3S

2021-01-24T22:40:32.169Z

Since literally everbody I know is currently working on a cloud@home project in some form I also decided to start working a long planned project: a portable kubernetes cluster.

And also a former colleague gave me the last push for it - thanks 😉

Portability

Currently, portability means, that only 2 cables - power and network - are connected.

A later addition I am planning is attaching the cluster to a mobile router. The question ist, if the POE switch can deliver the necessary additional power for this.

And, the 52Pi rack tower has a handle to carry it around - or throw it out of the window - depends on the use case.

This will become a series of articles, lets start with the hardware.

Hardware

Parts List

#	Part	each
4	Raspberry Pi 4 8GB Model B (ARMv8)	CHF 89.90
4	Verbatim Nano 64GB USB Sticks	CHF 11.95
1	52Pi ZP-0088 Rack Tower	CHF 35.00
1	POE Switch Tp-Link TL-SG1005P	CHF 51.30
4	POE Cables LogiLink (U/FTP, Kat. 6a, 25cm)	CHF 5.90
3	POE Hat Type U6109	CHF 23.90
1	POE Hat Type U6110	CHF 25.90
1	Dupont Pin Row	CHF 0.05
	Total	CHF 614.95

Dupont Pins on the U6109

The U6109 actually has two rows for connectors, mine had no connectors soldered in - unfortunately.

My soldering skills are a little dated, but I took my chances - and it worked out nicely.

Fan and LED connector cables can now easily be attached to the POE powered Raspberry.

Speaking of ventilation: the 52Pi rack tower ventilator is suprisingly quiet, but powerfull - so having it in the same room is not disturbing at all.

I removed the fan from the U6110, because they make a quite annoying sound.

Accessible SD Cards

A quite handy feature of the 52Pi rack tower are the SD card adapters. The card slots would be very hard to reach, when the case is fully assembled.

Mount the POE Switch

A simple wooden plate with a bunch of holes is connected to the base of the 52Pi rack tower using 4 blue cable ties.

The 2 Screws in the middle are connected the wall-mount recess of the POE Switch.

An acrylic plate would be nice, but I don’t have access to a laser-cutter.

Fully assembled

The fully assembled case with the 4 USB Sticks for the Storage Cluster.

Looks quite shiny by night 🤓

Portable Kubernetes Cluster based on Raspberry Pi 4 and Rancher K3S (this)
Setup mini Kubernetes Rancher K3S on Raspberry OS Lite
Storage class and nfs provisioner
Setting up a four times redundant gluster volume with mini USB sticks
Automatically provision Gluster volumes with Heketi

Gatsby Plugin Sharp not building on Raspberry PI (arm64)

2021-01-08T22:40:32.169Z

The gatsby plugin Sharp depends on Libvips and for some reason the Dockerfile.production from gatsby-starter-lumen it does not build on my Raspberry Pi 4.

make: Entering directory '/usr/src/app/node_modules/sharp/build'
  CC(target) Release/obj.target/nothing/../node-addon-api/nothing.o
  AR(target) Release/obj.target/../node-addon-api/nothing.a
  COPY Release/nothing.a
  TOUCH Release/obj.target/libvips-cpp.stamp
  CXX(target) Release/obj.target/sharp/src/common.o
../src/common.cc:24:10: fatal error: vips/vips8: No such file or directory
   24 | #include <vips/vips8>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [sharp.target.mk:137: Release/obj.target/sharp/src/common.o] Error 1
make: Leaving directory '/usr/src/app/node_modules/sharp/build'

I am pretty shure, that there are other ways, but adding a global package vips-dev solved the problem:

RUN apk add --update --no-cache --repository http://dl-3.alpinelinux.org/alpine/edge/community --repository http://dl-3.alpinelinux.org/alpine/edge/main vips-dev

The build takes very long (44 minutes), since literally everthing is compiled from scratch.

But this can be improved by using Docker caching or probably a base image.

More to come!

Kubernetes Version 1.20.0 breaks nfs provisioner

2021-01-06T22:40:32.169Z

Kubernetes version 1.20.0 breaks the nfs-subdir-external-provisioner.

The PersistentVolumeClaim stays in status “Pending”:

$ k get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Pending                                      managed-nfs-storage   10h

The pod logs of client provisioner contains:

unexpected error getting claim reference: selfLink was empty, can’t make reference

$ kubectl logs nfs-client-provisioner
E0106 06:18:07.328905       1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

unexpected error getting claim reference: selfLink was empty, can't make reference

The deprecation of selfLinks in Kubernetes 1.20.0 is responsible for this.

As noted by nealman13 in his github comment setting the feature gate RemoveSelfLink to false is a workaround for this problem.

But Rancher K3S doesn’t come with a kube-apiserver.yaml file. But feature gates can be passed as commandline arguments directly to the service.

For a fresh install of K3S the following commandline can be used:

$ curl -sfL https://get.k3s.io | sh -s - --kube-apiserver-arg "feature-gates=RemoveSelfLink=false"

An existing installation can be adjusted by changing the service directly:

$ nano /etc/systemd/system/k3s.service
... 
ExecStart=/usr/local/bin/k3s \
    server \
        '--kube-apiserver-arg' \
        'feature-gates=RemoveSelfLink=false' \
...

Contributing to an opensource project

2018-06-12T22:40:32.169Z

Only a few lines of code, but at the end I felt thrilled that I could contribute to the OfficeDevPnp project.

The steps are quite simple:

Fork the repo
Create your local dev environment
Implement your feature
Read the Contribution Guide
Fill out the paperwork
Open an upstream Pull Request
And probably get it merged

And the end you also get some kudos in the community blog

Thank you for your assistance and contributions on the behalf of the community. You are making a difference!

You’re welcome! 😁

Autofac and Automapper - A perfect match

2017-03-11T22:40:32.169Z

With hexagonal and onion architectures, clean code, domain-driven-design, command query responsibility segragation Dependency Injection or Inversion of Control comes in very handy.

One thing I came across while working with Autofac and Automapper is, how good the play together.

Imagine a basic setup where we handle an incoming request viewmodel, map it to an input command for a command handler, but the mapping is a little bit more complicated and we already want to use a service from the container during the mapping process. Automapper provides us with ITypeConverter and IValueResolver interfaces for more advanced mapping tasks.

For a mapping like this


var target = _mapper.Map<TargetType>(new SourceType());

We could use the following SourceType to TargetType TypeConverter

public class SourceToTargetTypeConverter 
                    : ITypeConverter<SourceType, TargetType>
{
    public TargetType Convert(SourceType source, TargetType destination, ResolutionContext context)
    {
        ...
        return targetObject;
    }
}

It could be quite handy to have access to the Autofac container and request some IFancyService inside our SourceToTargetTypeConverter

public class SourceToTargetTypeConverter 
                    : ITypeConverter<SourceType, TargetType>
{
    private readonly IFancyService _service;

    public SourceToTargetTypeConverter(IFancyService service) {
        _service = service;  // require IFancyService from the container
    }

    public TargetType Convert(
                  SourceType source, 
                  TargetType destination, 
                  ResolutionContext context)
    {
        return new TargetType
            {
                // use the service
                Thing = _service.GetAwesomeThing() 
            };
    }
}

For this we first need to create a MappingProfile that configures our SourceToTargetTypeConverter

public class FancyProfile : AutoMapper.Profile
{
  public FancyProfile()
  {
    CreateMap<SourceType, TargetType>()
      .ConvertUsing<SourceToTargetTypeConverter>();
  }
}

Later, when the container gets built we can use the assembly scanning feature of Autofac and register all AutoMapper.Profile types auomatically.

internal class MapperModule : Autofac.Module
{
    
  protected override void Load(ContainerBuilder builder)
  {

    builder.RegisterAssemblyTypes(Assembly.GetExecutingAssembly())
              .AssignableTo<Profile>()
              .As<Profile>()
              .AutoActivate();

    ...
  }
}

Now comes the magic trick: everytime when a mapper is resolved, we need to resolve a MapperConfiguration first. Because all AutoMapper.Profile types have been already registered in the container before we are now able to get all AutoMapper.Profile by resolving context.Resolve<IEnumerable<Profile>>() - another nice feature of Autofac.

When the actual Mapper is then created using the config.CreateMapper(Func<Type, object> serviceCtor) overload, Autofac will fulfill all dependencies defined in tha Automapper converters.


  // Get all Profiles and add them to the MapperConfiguration
  builder.Register(context =>
  {
      var profiles = context.Resolve<IEnumerable<Profile>>();
      var config = new MapperConfiguration(x =>
      {
        foreach (var profile in profiles)
        {
          x.AddProfile(profile);
        }
      });

      return config;
  }).SingleInstance().AutoActivate().AsSelf();


  // Resolve the MapperConfiguration and call CreateMapper()  
  builder.Register(context =>
  {
    var ctx = context.Resolve<IComponentContext>();
    var config = ctx.Resolve<MapperConfiguration>();
    return config.CreateMapper(
      t => ctx.Resolve(t) // The magic is happening here: Autofac will now fulfill all dependcies defined in MapperConfigurations
    );
  });

Works pretty nice that way and it perfectly fits into an hexagonal or onion architecture.

Btw, thanks Mario for pointing me to this idea 😘

Update 2021:

New automapper version
Find the updated code in the Github repo

Office 365 Junk Mail default setting

2014-08-06T22:40:32.169Z

Office 365 keeps filtering mail from recipients off my contact list in the “Junk E-Mail” folder which is quite annoying.

This behaviour can be changed via the “Settings > Options > Block or Allow”. By default “Trust email from my contacts” is turned off which is not the best choice for this setting.

SharePoint 2013 Composite Handbook

2013-02-18T22:40:32.169Z

For people like me who strive to use SharePoint 2013 oob features in their solutions, the SharePoint 2013 Composites Handbook is available for download here:

SharePoint 2013 Composites Handbook

Load and Execute for the SharePoint Client Object Model

2013-02-11T22:40:32.169Z

Since I am very lazy programmer I always think about shortening things a little bit.

Instead of writing…

Microsoft.SharePoint.Client.Web appWeb = siteContext.Web;
siteContext.Load(appWeb);
siteContext.ExecuteQuery();

… its worth having a look on the signature of the Load method:

public void Load<T>(T clientObject, params Expression<Func<T, object>>[] retrievals) where T : ClientObject;

It turns out that I can save one line of code every time by declaring an extension method like…

public static void LoadAndExecute<T>(
  this Microsoft.SharePoint.Client.ClientContext me,
  T clientObject,
  params System.Linq.Expressions.Expression<Func<T, object>>[] retrievals) 
         where T : Microsoft.SharePoint.Client.ClientObject
  {
    me.Load(clientObject, retrievals);
    me.ExecuteQuery();
  }

which simplifies the above code to:

Microsoft.SharePoint.Client.Web appWeb = siteContext.Web;
siteContext.LoadAndExecute(appWeb);

One line saved…

Create a SharePoint 2013 Workflow with a custom Initiation Form in Visual Studio 2012

2013-02-07T22:40:32.169Z

Prerequisites

Create a on-prem high trust app as described in my previous blog post.

Add the Workflow

First of all we need to add a list to host the workflow by adding a new item of type “List” to the SharePoint AppWeb project.

In a further step, the list could be configured using the new list designer in Visual Studio 2012.

Now that we have a list, we can add a workflow to be run on that list by adding a new Item of type “Workflow”to the SharePoint AppWeb project. Set the workflow to start manually and associate it with the previously created list. Select the according items to create a Workflow History And Workflow Tasks List for the Workflow.

The new Workflow Designer should now be visible showing a workflow with an empty sequence.

The custom initiation form will just contain a single string. In order to use that string within the workflow a variable has to be added as input parameter, e.g. “inputString”. To show that the value is actually submitted to the workflow we use a “WriteToLog” Activity that writes the value of our input parameter to the Workflow History List.

The next step is to add a custom Initiation Form by adding a new Item of Type “InitiatonForm” to the Workflow element. This adds a default InitiationForm to the project which is already populated with some code.

Clean up the form to only contain a single text field for our string input variable. In the “startWorkflow” function triggered by the submit button the value of the text field must be added to the params object. It is also necessary to get the List Item ID from the URL, since this is needed as parameter for the “startWorkflowOnListItem” method on the “WorkflowInstance” object.

The last step is to add a reference to the custom initiation form in the Elements deklaration of the workflow element.

The interesting part here is, that the URL Token replacement does not seem to work properly here as I posted in Apps for SharePoint MSDN forum. This means that custom initiation and association forms can only be created in the AppWeb and not in the Remote App, which makes more complicated scenarios pretty much bound to javascript/html implementations for now.

After a deploy the ListItem has a workflow associated to it and after starting it manually, the user gets our custom initiation form presented.

After the workflow is completed, the workflow history log shows the value entered in the initiation form.

2013 Workflow Initiation Form in ~remoteAppUrl

Creating a SharePoint 2013 Provider Hosted on-prem App

2013-02-07T22:40:32.169Z

Prerequisites

First of all a on-prem development environment has to be created. I used the How to: Set up an on-premises development environment for apps for SharePoint found in MSDN.

High Trust App

In order to create an on-prem app, the app must be registered in SharePoint as “trusted token issuer”. There are currently two different procedures depending on the version of Office Developer Tools for Visual Studio 2012. The respective procedures by using self-signed certificates is described in How to: Create high-trust apps for SharePoint 2013 using the server-to-server protocol (advanced topic).

I am using Active Directory Certificate services my development environment, so I am able to create certificate with my own trusted Root Authority in the IIS configuration UI by clicking “Create Domain Certificate”.

After creating the certificate, both the public (.cer) and the private (.pfx) key have to be exported. The .cer file is exported by opening the certificate and hitting “Copy to file” on the “Details” tab. Use the default settings in the dialog.

The .pfx file can be exported by clicking the “Export” in the Actions sidebar. Don’t forget to set a password, since it is needed later when creating the Visual Studio project.

The next step is to register the App as trusted token issuer, I streamlined this process in a PowerShell script for the Preview 2 Version of the Office Developer Tools for Visual Studio 2012. The first parameter of the script denotes the site, where the app will be deployed to and the second parameter points to the public key (.cer) file.

The result shows a new IssuerID which is used to identify the app. The IssuerID has to be entered when creating the Visual Studio Project.

After creating a new App for SharePoint 2013 Project, select Provider-Hosted as hosting option. Enter a name and the site url used before and hit next.

Select “Use a certificate” for authentication and select the private key (.pfx) file, enter the IssuerID created by the PowerShell script before and the password for the private key file.

Hit F5 if you get a page showing the Title of the HostWeb everythings OK 🙂

Add new Trusted Token Issuer to a SharePoint 2013 Site – S2S HighTrust Apps

2012-12-18T22:40:32.169Z

For SharePoint 2013 App Development on-premise, there might not be a Cloud ACS around to get the token stuff done. For this case the App must create its own access tokens and SharePoint 2013 must know about them as Trusted Security Token Issuers. The App creates a token (with the users credentials) and signs it with its own private key. If the SharePoint 2013 Site hosting the app knows about the IssuerID, the Server can extract the token and grant access.

The whole process is described in How to: Create high-trust apps for SharePoint 2013 using the server-to-server protocol (advanced topic). Note that there is a difference between SharePoint Developer Tools Preview and Preview 2.

Based on Andrew Connell’s Script for the Beta and RTM Version, I composed a little powershell script that streamlines the registration, since this has to be done for every App. All that is needed is a developer site and a private certificate.

Update: The flag –IsTrustBroker in New-SPTrustedSecurityTokenIssuer makes the certificate reusable. So several apps can share the same certificate ussing the same issuer ID. [0] – Bullet: Token issuer not configured as trust broker.

# ———————————————————————————
# Setup for a new Trusted Security Token Issuer for an on-prem Developer Hosted App
# $TargetUrl = Developer SiteCollection Url
# $publicCertPath = Path to the public part of the certificate
# (optional) $issuerID = Unique ID for the App, generated if omitted
#
# e.g. .\AddTrustedTokenIssuerToSite “http://developerSite/” “c:\PathTo.Cer”
#
# The isserID is printed at the end of the script -> keep it for
# the App’s web.config
# ———————————————————————————

param(
[string]$TargetUrl,
[string]$publicCertPath,
[string]$issuerID = “”
)

Write-Host
Write-Host “Register new Trusted Identity Token Issuer…” -ForegroundColor White
Write-Host ” Script Steps:” -ForegroundColor White
Write-Host ” (1 of 5) Validating parameters…” -ForegroundColor White
Write-Host ” (2 of 5) Verify SharePoint PowerShell Snapin Loaded” -ForegroundColor White
Write-Host ” (3 of 5) Check if site collection exists at target $SiteUrl” -ForegroundColor White
Write-Host ” (4 of 5) Adding IssuerIdentifier at target $SiteUrl” -ForegroundColor White
Write-Host ” (5 of 5) Verify OAuth over https is disabled” -ForegroundColor White
Write-Host

# ———————————————–

# verify parameters passed in
Write-Host “(1 of 5) Validating parameters…” -ForegroundColor White
if ($TargetUrl -eq $null -xor $TargetUrl -eq “”) {
  Write-Error ‘$TargetUrl is required’
  Exit
}
if ($publicCertPath -eq $null -xor $publicCertPath -eq “”) {
  Write-Error ‘$publicCertPath is required’
  Exit
}
Write-Host ” All parameters valid” -ForegroundColor Gray

# ———————————————–

if ($issuerID -eq $null -xor $issuerID -eq “”) {
  Write-Host ‘ IssuerID is empty, generating’
  $issuerID = [System.Guid]::NewGuid().ToString()
}

# ———————————————–

# Load SharePoint PowerShell snapin
Write-Host
Write-Host “(2 of 5) Verify SharePoint PowerShell Snapin Loaded” -ForegroundColor White
$snapin = Get-PSSnapin | Where-Object {$_.Name -eq ‘Microsoft.SharePoint.PowerShell’}
if ($snapin -eq $null) {
  Write-Host ” .. loading SharePoint PowerShell Snapin…” -ForegroundColor Gray
  Add-PSSnapin “Microsoft.SharePoint.PowerShell”
}
Write-Host ” Microsoft SharePoint PowerShell snapin loaded” -ForegroundColor Gray

# ———————————————–

Write-Host
Write-Host “(3 of 5) Check if site collection exists at target $TargetUrl” -ForegroundColor White
$targetSpSite = Get-SPSite $TargetUrl
if ($targetSpSite -ne $null) {
  Write-Host ” … site found” -ForegroundColor Gray

  $realm = Get-SPAuthenticationRealm -ServiceContext $targetSpSite
  $certificate = Get-PfxCertificate $publicCertPath
  $fullIssuerIdentifier = $issuerId + ‘@’ + $realm

  Write-Host
  Write-Host “(4 of 5) Adding IssuerIdentifier $fullIssuerIdentifier ” -ForegroundColor White
  Write-Host ” as TrustedSecurityTokenIssuer to $TargetUrl” -ForegroundColor White

  $result = New-SPTrustedSecurityTokenIssuer -Name $issuerId -Certificate $certificate -RegisteredIssuerName $fullIssuerIdentifier –IsTrustBroker

  Write-Host
  write-host “— ### Results ### —” -ForegroundColor Green
  write-host ” Realm: $realm” -ForegroundColor Green
  # write-host “FullIssuerID: $fullIssuerIdentifier” -ForegroundColor Green
  write-host ” IssuerID: $issuerId” -ForegroundColor Green
  write-host ” Result: $result” -ForegroundColor Green
  write-host “— ### Results ### —” -ForegroundColor Green
}
else
{
  Write-Error “$TargetUrl not found”
  Exit
}

Write-Host
Write-Host “(5 of 5) Verify OAuth over https is disabled” -ForegroundColor White
$serviceConfig = Get-SPSecurityTokenServiceConfig
if($serviceConfig.AllowOAuthOverHttp -ne $true) {
    $serviceConfig.AllowOAuthOverHttp = $true
    $serviceConfig.Update()
    write-host ‘ OAuth over https disabled’ -ForegroundColor Green
}
else
{
  write-host ‘ OAuth over https is already disabled’ -ForegroundColor White
}

0: More TroubleShooting Tips for High Trust Apps on SharePoint 2013

Export Farm Solution wsp Files SharePoint 2007 and 2010

2012-06-04T22:40:32.169Z

The following PowerShell Snippet exports all SharePoint Farm Solution Packages (.wsp) from the farm solution store. It does work on Windows 2003 R2 with PowerShell 1.0, which I find very practical because it is the standard configuration for WSS3 and MOSS 2007 installations.

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$farm = [Microsoft.SharePoint.Administration.SPFarm]::Local
$farm.Solutions | % {
  $filename = ($pwd.ToString() + "\" + $_.SolutionFile.Name);
  write-host ("Saving" + $filename);
  $_.SolutionFile.SaveAs($filename)
}

From: MSDN

Martin Loitzl's Blog

Lose a Node, Keep the Service: Building a Zero-Downtime 3-Node k3s Cluster

The Threat Model

Layer 1: k3s Control Plane with etcd

Layer 2: Distributed Storage with Longhorn

Layer 3: MongoDB Replica Set

Layer 4: Typesense Raft Cluster

Layer 5: Stateless Pods — the Topology Spread Trap

What Surviving Node Loss Actually Looks Like

The Numbers

Is This Overkill for a Homelab?

The Greenfield Sandbox: Implementing Hard Tenant Isolation and Deterministic Data at Scale

1. Physical Isolation: Why tenant_id Isn’t Enough

Dynamic Database Routing

The Indexing Challenge

2. The Blended Scoring Engine: Achieving Determinism

3. The BFF Pattern: Security by Design

JWT Hiding

4. Scaling the Frontend: Strict Relay & Atomic Design

Fragment Colocation

“Render-as-you-Fetch”

The “Silent” Tenant-Switch Bug

The Lessons Learned

The Stack Summary:

Variance and covariance explained with C#

The Gentle Theory Bit

Our Animal Hierarchy

Covariance With out

Async Streams (IAsyncEnumerable<out T>)

Read-Only Collections

Async LINQ Extensions

Contravariance With in

Dependency Injection And Contravariance

Comparers and Dependency Injection

The Classic Pitfalls

Why isn’t Everything Covariant or Contravariant?

Arrays are covariant

Variance In JavaScript And TypeScript

Conclusion

Deleting deleted group connected SharePoint Online Sites

Getting an App Only access token for SharePoint REST APIs

Objectives

App Registration

Getting an OAuth2 Token using a Client Assertion

The jwtgen Tool

Everything put together

Getting all Lists from a SharePoint Site using the REST API

Using a client secret

Structure of the Bearer Token

Conclusion

References

Getting a Term using a Custom Property Value with PnP Core

Objective

Retrieving Terms using custom properties

Retrieving Azure Application Insights using Azure CLI

Objective

Authentication Options

Interactive Login

Automated Login

Find the Azure Insights Resource

Retrieving Events

Executing Analytics Queries

Correlate Log entries

Outlook

Syncing MS Graph Properties with SharePoint User Profile from Azure Function

Introduction

Creating the User Profile Property

Creating the Search Managed Property

Setup Authentication for the Azure Function

The Azure Function

The Problem

Enabling App Only Authentication

Results

Final Notes

Troubleshooting SharePoint Online Search

Introduction

Analyze Crawl Log

Re-crawl a Document Library

Managed Properties

Search Query Tool

1. Physical Isolation: Why `tenant_id` Isn’t Enough

Covariance With `out`

Async Streams (`IAsyncEnumerable<out T>`)

Contravariance With `in`

The `jwtgen` Tool