https://abdus.dev personal website of Abdussamet Kocak kopkop Wed, 21 Oct 2026 00:00:00 +0000 Wed, 21 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/demotime/ https://abdus.dev/projects/demotime/ <p>I've built this app for myself to trim down demo videos by cutting off sections where nothing happens, e.g. when waiting for a command to finish.</p> <p><img src="/projects/demotime/./demotime.png" alt="demotime web app screenshot" /></p> <p>I could use a video editor, but I don't want to spend time picking out the sections manually. Turns out ffmpeg has a filter for this. So I first built a Python script, then I quickly realized that this could be better off as a web app.</p> <a class="repo-card" href="https://github.com/abdusco/demotime" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>demotime</strong> </span> <span class="repo-card__description">A minimal web app for trimming still sections off of a video</span> <span class="repo-card__meta"> <span>JavaScript</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>The app is deployed as a PWA, so you can install it on your device. It's available at <a href="https://demotime.pages.dev">https://demotime.pages.dev</a></p> Tue, 20 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/subbed/ https://abdus.dev/projects/subbed/ <p>I've created this app to share videos with my friends who don't speak English. It lets you import a Youtube video, transcribe its audio into an SRT subtitle file, then translate it into different languages.</p> <a class="repo-card" href="https://github.com/abdusco/subbed" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>subbed</strong> </span> <span class="repo-card__meta"> <span>Go</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>It has a built-in player that overlays the subtitles on top of the video, so you can watch it as if the original creator had provided subtitles in your language.</p> <p>It uses <a href="https://github.com/yt-dlp/yt-dlp" target="_blank" rel="noopener"><code>yt-dlp</code></a> under the hood. So you have to copy over your Youtube cookies to download the audio. You can also upgrade it to the latest nightly build when the existing version stops working.</p> <p>The app also lets you use custom models from OpenRouter to transcribe the audio and translate the subtitles.</p> <image-gallery class="image-gallery" role="region" aria-label="Media gallery"> <div class="image-gallery__track" tabindex="0" aria-label="Scroll to see more images and videos"> <img src="./subbed_login.png" alt="login screen for the app" /> <img src="./subbed_library.png" alt="library view showing available videos and subtitles" /> <img src="./subbed_menu.png" alt="settings menu" /> <img src="./subbed_settings.png" alt="settings screen" /> <img src="./subbed_add.png" alt="popup for adding new videos" /> <video src="./subbed_transcript.mp4" controls playsinline preload="metadata"></video> <video src="./subbed_translate.mp4" controls playsinline preload="metadata"></video> </div> </image-gallery> Tue, 13 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/wp/ https://abdus.dev/projects/wp/ <div class="app-logo"><img src="wp-icon.svg" alt="wp app icon"/></div> <style> .app-logo { width: 8rem; height: 8rem; margin-bottom: 2rem; } </style> <p>wp is a menu app that offers a minimal wallpaper switcher. It lets you cycle through your favorite wallpapers on a timeout or with a hotkey. It applies a blur effect and lets you reveal the image with double click on the desktop.</p> <a class="repo-card" href="https://github.com/abdusco/wp" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>wp</strong> </span> <span class="repo-card__description">A minimal macOS wallpaper switcher</span> <span class="repo-card__meta"> <span>Swift</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>It can also stack multiple portrait images side-by-side to create a collage wallpaper. It has a settings panel to configure the hotkeys, auto-cycle timeout, HTTP API port, and other options.</p> <image-gallery class="image-gallery" role="region" aria-label="wp screenshots"> <div class="image-gallery__track" tabindex="0" aria-label="Scroll to see more images and videos"> <img src="./wp-menu.png" alt="wp menu app screenshot" loading="lazy"> <img src="./wp-settings.png" alt="wp settings panel screenshot" loading="lazy"> <img src="./wp-sources.png" alt="wp settings panel screenshot" loading="lazy"> <img src="./wp-hotkeys.png" alt="wp settings panel screenshot" loading="lazy"> </div> </image-gallery> <p>The HTTP server is meant for remote control, which you can activate by specifying a <code>PORT</code> environment variable or in the settings panel. You can use <code>curl</code> to send commands to it e.g. with Alfred on a hotkey press.</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash"><span class="z-nv">PORT</span><span class="z-o">=</span><span class="z-m">9876</span> ~/bin/wp ~/Pictures/wallpapers/ curl localhost:9876/next <span class="z-c1"># Next image</span> curl localhost:9876/prev <span class="z-c1"># Previous image</span> curl localhost:9876/blur <span class="z-c1"># Toggle blur</span></code></pre> Mon, 12 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/tricklebar/ https://abdus.dev/projects/tricklebar/ <div class="app-logo"><img src="./tricklebar-icon.svg" alt="TrickleBar Logo"></div> <style> .app-logo { width: 8rem; height: 8rem; margin-bottom: 2rem; } </style> <p>TrickleBar is a menu app based on <a href="https://aria2.github.io/" target="_blank" rel="noopener">aria2</a> that provides a minimal interface to view and manage your downloads. It requires aria2 to be installed and available in <code>$PATH</code> or other common locations, but you can also override which aria2 binary to use. It has a minimal settings panel to configure aria2 options.</p> <image-gallery class="image-gallery" role="region" aria-label="tricklebar screenshots"> <div class="image-gallery__track" tabindex="0" aria-label="Scroll to see more images and videos"> <img src="./tricklebar-popup.png" alt="tricklebar menu app screenshot" loading="lazy"> <img src="./tricklebar-settings.png" alt="tricklebar settings panel screenshot" loading="lazy"> </div> </image-gallery> <a class="repo-card" href="https://github.com/abdusco/tricklebar" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>tricklebar</strong> </span> <span class="repo-card__description">A lightweight macOS menu bar download manager, backed by aria2c.</span> <span class="repo-card__meta"> <span>Swift</span> <span aria-label="1 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 1 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>It also provides a CLI that allows you to add new downloads, and passes the extra switches to aria2c.</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">&gt; TrickleBar.app/Contents/MacOS/TrickleBar --help tricklebar — aria2c-compatible download manager Usage: tricklebar <span class="z-o">[</span>options<span class="z-o">]</span> URL... Add download<span class="z-o">(</span>s<span class="z-o">)</span> to running daemon tricklebar -i FILE Read URLs from file open TrickleBar.app Launch menu bar interface Options <span class="z-o">(</span>subset of aria2c flags<span class="z-o">)</span>: -d, --dir<span class="z-o">=</span>DIR Download directory -o, --out<span class="z-o">=</span>FILE Output filename -s, --split<span class="z-o">=</span>N Connections per download -x, --max-connection-per-server<span class="z-o">=</span>N ... -h, --help -v, --version</code></pre> <p>It listens to <code>tricklebar://</code> URLs to add downloads. You can even specify which script to run when a download is done. I use it to automatically move the downloaded file to the right folder and show a notification, for example.</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash"><span class="z-nv">url</span><span class="z-o">=</span><span class="z-s2">&#34;</span><span class="z-nv">$1</span><span class="z-s2">&#34;</span> <span class="z-nv">encoded</span><span class="z-o">=</span><span class="z-k">$(</span>python3 -c <span class="z-s2">&#34;import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1]))&#34;</span> <span class="z-s2">&#34;</span><span class="z-nv">$url</span><span class="z-s2">&#34;</span><span class="z-k">)</span> open <span class="z-s2">&#34;tricklebar://add-download?on_done=</span><span class="z-nv">$HOME</span><span class="z-s2">/bin/aria2_on_done.py&amp;url=</span><span class="z-nv">$encoded</span><span class="z-s2">&#34;</span></code></pre> <p>You can download the <a href="https://github.com/abdusco/tricklebar/releases/latest" target="_blank" rel="noopener">latest version</a> from GitHub.</p> Sun, 11 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/kuller/ https://abdus.dev/projects/kuller/ <div class="app-logo"><img src="./kuller-icon.svg" alt="kuller app icon"></div> <style> .app-logo { width: 8rem; height: 8rem; margin-bottom: 2rem; } </style> <p>kuller is an image culling app to quickly weed out bad photos and keep the best ones. It is designed to be minimal. It has a canvas window like Photoshop, and a side panel with a filmstrip of all the images in the current folder. You can quickly go through the images and mark them as keep or discard, or crop them in various aspect ratios using hotkeys.</p> <video src=".&#x2F;kuller_demo.mp4" poster=".&#x2F;kuller_demo.jpg" width="1626" height="1080" preload="none" onclick="if (!this.played.length && this.paused) this.play().catch(console.error)" controls muted playsinline ></video> <a class="repo-card" href="https://github.com/abdusco/kuller" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>kuller</strong> </span> <span class="repo-card__description">Minimal image culling app for macOS</span> <span class="repo-card__meta"> <span>Swift</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>I built it to get ahead of my ever-growing pile of photos and I didn't like Lightroom's interface to do it. The first iteration was built as a web app backed by golang. I found myself using it quote often, so I decided to turn it into a native macOS app.</p> <p>It has a minimal interface, and a settings panel to customize the hotkeys. It offers a CLI interface which is how I use it.</p> <p>You can download <a href="https://github.com/abdusco/kuller/releases/latest" target="_blank" rel="noopener">the latest version</a> on GitHub.</p> Sat, 10 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/golang-database-tests/ https://abdus.dev/posts/golang-database-tests/ <p>You can have a test for a method that executes an SQL query, but it won't tell you if the query is actually correct. Seeing a 100% coverage for it is a false assurance. You actually have to test your query against a real database.</p> <h2 id="preparing-the-database">Preparing the database</h2> <p>When I worked with Django, it offered a built-in test framework that <a href="https://docs.djangoproject.com/en/dev/topics/testing/tools/#django.test.TransactionTestCase.fixtures" target="_blank" rel="noopener">runs tests against a temporary database</a>. It created the database, ran the migrations, and then ran the tests. They could even be isolated within a transaction without causing bad interactions.</p> <p>For Django there was even a sidekick library called <a href="https://factoryboy.readthedocs.io/en/stable/" target="_blank" rel="noopener">factory_boy</a>, that would hook into the ORM and create the right objects with the right relations and plug in fake data where needed. You'd be able to override a field nested inside a relation to make your assertions work.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-c1"># this creates a person, </span> <span class="z-c1"># and a related address with the city set to &#39;New York&#39;.</span> <span class="z-n">p</span> <span class="z-o">=</span> <span class="z-n">PersonFactory</span><span class="z-o">.</span><span class="z-n">create</span><span class="z-p">(</span> <span class="z-n">name</span><span class="z-o">=</span><span class="z-s2">&#34;John Doe&#34;</span><span class="z-p">,</span> <span class="z-n">address__city</span><span class="z-o">=</span><span class="z-s2">&#34;New York&#34;</span><span class="z-p">,</span> <span class="z-p">)</span> <span class="z-k">assert</span> <span class="z-n">p</span><span class="z-o">.</span><span class="z-n">address</span><span class="z-o">.</span><span class="z-n">city</span> <span class="z-o">==</span> <span class="z-s2">&#34;New York&#34;</span></code></pre> <p>There are some solutions for database testing in Golang, such as <a href="https://github.com/DATA-DOG/go-sqlmock" target="_blank" rel="noopener">sqlmock</a> or <a href="https://github.com/testcontainers/testcontainers-go" target="_blank" rel="noopener">testcontainers</a>, but they all feel clunky, cumbersome to use because of one thing: fixtures. How do you populate the database with the right data for your tests?</p> <p>ORMs are not welcome in Golang. The language is not really built for dynamism. We write SQL statements directly without using an ORM, so the relations are only mapped in the developer's head. We can accept this as a fact and build our harness around it.</p> <p>The <a href="https://github.com/go-testfixtures/testfixtures" target="_blank" rel="noopener">testfixtures</a> library helps with this. You write YAML files that describe which tables have which rows, and the library will insert those rows.</p> <pre data-lang="yaml" data-highlighted="true" class="language-yaml z-code z-chroma"><code class="language-yaml" data-lang="yaml"><span class="z-nt">persons</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-w"> </span>- <span class="z-nt">id</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-m">1</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">name</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">John Doe</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">address_id</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-m">21</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">addresses</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-w"> </span>- <span class="z-nt">id</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-m">21</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">city</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">Berlin</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">street</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">Boxhagenstr. 10</span><span class="z-w"> </span></code></pre> <p>We can build a test helper that populates the database with the right data and runs the queries against it.</p> <pre data-lang="go" data-highlighted="true" class="language-go z-code z-chroma"><code class="language-go" data-lang="go"><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">applyFixtures</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">fixturePath</span><span class="z-w"> </span><span class="z-kt">string</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Helper</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">fixtures</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">testfixtures</span><span class="z-p">.</span><span class="z-nf">New</span><span class="z-p">(</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">testfixtures</span><span class="z-p">.</span><span class="z-nf">Database</span><span class="z-p">(</span><span class="z-nx">db</span><span class="z-p">),</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">testfixtures</span><span class="z-p">.</span><span class="z-nf">Dialect</span><span class="z-p">(</span><span class="z-s">&#34;postgres&#34;</span><span class="z-p">),</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">testfixtures</span><span class="z-p">.</span><span class="z-nf">Files</span><span class="z-p">(</span><span class="z-nx">fixturePath</span><span class="z-p">),</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">return</span><span class="z-w"> </span><span class="z-nx">fixtures</span><span class="z-p">.</span><span class="z-nf">Load</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">newTestDB</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// this is a database created from the latest seed</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nf">Open</span><span class="z-p">(</span><span class="z-s">&#34;postgres&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;user=postgres dbname=test sslmode=disable&#34;</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Cleanup</span><span class="z-p">(</span><span class="z-kd">func</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">.</span><span class="z-nf">Close</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-p">})</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">return</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">TestGetUserAddress</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nf">newTestDB</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nf">applyFixtures</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;testdata/get_user_address.yaml&#34;</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">repo</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nf">NewRepo</span><span class="z-p">(</span><span class="z-nx">db</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">userID</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-mi">1</span><span class="z-w"> </span><span class="z-c1">// this refers to an ID in the fixture</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">addr</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">repo</span><span class="z-p">.</span><span class="z-nf">GetUserAddress</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Context</span><span class="z-p">(),</span><span class="z-w"> </span><span class="z-nx">userID</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">assert</span><span class="z-p">.</span><span class="z-nf">Equal</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;Berlin&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">addr</span><span class="z-p">.</span><span class="z-nx">City</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span></code></pre> <h2 id="isolating-tests">Isolating tests</h2> <p>This is nice and all, but it leaves the database in a dirty state. Those rows remain in the database, and if we run multiple tests, they might interfere with each other. We need to wrap each test in a transaction and roll it back when the test ends.</p> <p>The <a href="https://github.com/DATA-DOG/go-txdb" target="_blank" rel="noopener">txdb</a> library fits well here.</p> <pre data-lang="go" data-highlighted="true" class="language-go z-code z-chroma"><code class="language-go" data-lang="go"><span class="z-kn">import</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-o">...</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-s">&#34;github.com/DATA-DOG/go-txdb&#34;</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">_</span><span class="z-w"> </span><span class="z-s">&#34;github.com/lib/pq&#34;</span><span class="z-w"> </span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">init</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// we register an SQL driver named &#34;txdb&#34;</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">txdb</span><span class="z-p">.</span><span class="z-nf">Register</span><span class="z-p">(</span><span class="z-s">&#34;txdb&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;postgres&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;user=postgres dbname=test sslmode=disable&#34;</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">newTestDB</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// this is a database created from the latest seed</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">realDB</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nf">Open</span><span class="z-p">(</span><span class="z-s">&#34;postgres&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;user=postgres dbname=test sslmode=disable&#34;</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// apply the migrations to prepare it for tests</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">migrations</span><span class="z-p">.</span><span class="z-nf">Run</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Context</span><span class="z-p">(),</span><span class="z-w"> </span><span class="z-nx">realDB</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// use a dynamic name for the database connection to avoid conflicts</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">testDB</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nf">Open</span><span class="z-p">(</span><span class="z-s">&#34;txdb&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Name</span><span class="z-p">())</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Cleanup</span><span class="z-p">(</span><span class="z-kd">func</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">testDB</span><span class="z-p">.</span><span class="z-nf">Close</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-p">})</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">return</span><span class="z-w"> </span><span class="z-nx">testDB</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span></code></pre> <h3 id="dealing-with-the-setup-overhead">Dealing with the setup overhead</h3> <p>Running migrations for every test means the schema version is checked each time. The logic in <code>migrations.Run</code> hopefully skips over applied migrations, but still, it adds overhead across multiple packages. We need to run it once per test run, which includes all packages being tested.</p> <p>We might reach for a <code>sync.Once</code>, but it won't work as you'd expect. Golang compiles your tests and executes them in parallel as separate processes. So the <code>sync.Once</code> will only work for a single test process, not across all of them.</p> <p>We need a global mutex to synchronize all those migrations. Postgres has our back.</p> <pre data-lang="go" data-highlighted="true" class="language-go z-code z-chroma"><code class="language-go" data-lang="go"><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">lock</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Helper</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">const</span><span class="z-w"> </span><span class="z-nx">lockID</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-mi">1234567890</span><span class="z-w"> </span><span class="z-c1">// some unique number</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">_</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">.</span><span class="z-nf">Exec</span><span class="z-p">(</span><span class="z-s">&#34;SELECT pg_advisory_lock($1)&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">lockID</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">return</span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">_</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">.</span><span class="z-nf">Exec</span><span class="z-p">(</span><span class="z-s">&#34;SELECT pg_advisory_unlock($1)&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">lockID</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span></code></pre> <p><code>lockID</code> is a hardcoded random number <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> that is shared across all runs. This removes the need for a <code>sync.Once</code> and allows us to run migrations once per test run.</p> <pre data-lang="go" data-highlighted="true" class="language-go z-code z-chroma"><code class="language-go" data-lang="go"><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">migrateTestDB</span><span class="z-p">(</span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">unlock</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nf">lock</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">defer</span><span class="z-w"> </span><span class="z-nf">unlock</span><span class="z-p">()</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// migrations will run once per test run.</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">migrations</span><span class="z-p">.</span><span class="z-nf">Run</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">.</span><span class="z-nf">Context</span><span class="z-p">(),</span><span class="z-w"> </span><span class="z-nx">db</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">newTestDB</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">testing</span><span class="z-p">.</span><span class="z-nx">T</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nx">DB</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// this is a database created from the latest seed</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">realDB</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">sql</span><span class="z-p">.</span><span class="z-nf">Open</span><span class="z-p">(</span><span class="z-s">&#34;postgres&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;user=postgres dbname=test sslmode=disable&#34;</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">require</span><span class="z-p">.</span><span class="z-nf">NoError</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">err</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nf">migrateTestDB</span><span class="z-p">(</span><span class="z-nx">t</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">realDB</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-o">...</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span></code></pre> <h2 id="where-it-falls-short">Where it falls short</h2> <p>This is a simplified version of what we use in production. However, it has some downsides.</p> <p>For one, the fixtures are not easy to prepare. Relations and values should make sense. Denormalized columns should be consistent across tables. We often end up with a lot of boilerplate data in the YAML files, because real-world tables don't consist of a handful of columns — they have tens of columns. Queries often join many tables, so you have to specify the supporting rows for your query <sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p> <p>Also, the queries are often not simple. If they have many <code>WHERE</code> clauses, it is not easy to prepare the &quot;right&quot; fixture that would satisfy the right combination of conditions. So you end up with huge fixtures that are difficult to maintain and reason about.</p> <p>The fixtures are also not reviewable, because you can't put them next to the test code that uses them. You often end up referring to records by their IDs, which are not meaningful. The reviewer has to take the author's word for it.</p> <p>This setup makes it straightforward to test your business logic that's embedded in SQL queries against a real database. The harder part is maintaining the fixtures, keeping them small enough that the test's intent remains clear.</p> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p><a href="https://xkcd.com/221/">https://xkcd.com/221/</a>&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>One way to reduce YAML bloat is to use YAML anchors, which allow you to define a table record once and reference it in multiple places. For example:</p> <pre data-lang="yaml" data-highlighted="true" class="language-yaml z-code z-chroma"><code class="language-yaml" data-lang="yaml"><span class="z-nt">addresses</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-w"> </span>- <span class="z-cp">&amp;address_props</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">id</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-m">21</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">city</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">Berlin</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">street</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">Boxhagenstr. 10</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c"># imagine we have 10 more columns...</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span>- <span class="z-nt">&lt;&lt;</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-cp">*address_props</span><span class="z-w"> </span><span class="z-c"># reuse the fields from the first record, override the ID and city</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">id</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-m">22</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nt">city</span><span class="z-p">:</span><span class="z-w"> </span><span class="z-l">Hamburg</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c"># street and other columns are left out because they&#39;re irrelevant for the test that references this address record.</span><span class="z-w"> </span></code></pre> &#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></li> </ol> </div> Sat, 10 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/htmlpopup/ https://abdus.dev/projects/htmlpopup/ <div class="htmlpopup-icon"><img src="./htmlpopup-icon.svg" alt="htmlpopup app icon"/></div> <style> .htmlpopup-icon { width: 8rem; height: 8rem; margin-bottom: 2rem; } </style> <p>htmlpopup is a simple utility for macOS that allows you to open up an HTML page in a popup window. It provides a JS API to control the window, interact with the filesystem, returning values to the caller on exit etc. You can call it with a URL, or a local HTML file, or a directory, or an HTML literal or pipe HTML to it. It supports <code>localStorage</code> as well.</p> <a class="repo-card" href="https://github.com/abdusco/htmlpopup" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>htmlpopup</strong> </span> <span class="repo-card__description">A simple macOS command-line utility to display HTML content or a URL in a native webview popup window.</span> <span class="repo-card__meta"> <span>Swift</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>I built it to show basic UIs for other utilities, e.g. one for having an AI chat window, or a grammar checker, or a UI in front of a UI. You can inject environment variables and other data into the webview (accessible via <code>window.env.*</code>).</p> <p>For example, this is how I use it for the AI chat:</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">htmlpopup --id chat --env.PROMPT <span class="z-s2">&#34;</span><span class="z-nv">$1</span><span class="z-s2">&#34;</span> <span class="z-se">\ </span> --env.GEMINI_API_KEY <span class="z-nv">$GEMINI_API_KEY</span> <span class="z-se">\ </span> --env.OPENAI_API_KEY <span class="z-nv">$OPENAI_API_KEY</span> <span class="z-se">\ </span> --env.ANTHROPIC_API_KEY <span class="z-nv">$ANTHROPIC_API_KEY</span> ~/dev/web/chat</code></pre> <p>where <code>$1</code> is the currently selected text (populated by Alfred on a hotkey press).</p> <p>In another use case, I wrap it in python to use an API + HTML served by that script:</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-nd">@contextlib.contextmanager</span> <span class="z-k">def</span> <span class="z-nf">run_htmlpopup</span><span class="z-p">(</span><span class="z-n">address</span><span class="z-p">:</span> <span class="z-nb">str</span><span class="z-p">):</span> <span class="z-n">exe_path</span> <span class="z-o">=</span> <span class="z-n">shutil</span><span class="z-o">.</span><span class="z-n">which</span><span class="z-p">(</span><span class="z-s2">&#34;htmlpopup&#34;</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-ow">not</span> <span class="z-n">exe_path</span><span class="z-p">:</span> <span class="z-k">raise</span> <span class="z-ne">RuntimeError</span><span class="z-p">(</span><span class="z-s2">&#34;htmlpopup executable not found in PATH.&#34;</span><span class="z-p">)</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">Popen</span><span class="z-p">([</span><span class="z-n">exe_path</span><span class="z-p">,</span> <span class="z-s2">&#34;--title&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;search&#34;</span><span class="z-p">,</span> <span class="z-n">address</span><span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">)</span> <span class="z-k">try</span><span class="z-p">:</span> <span class="z-k">yield</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">wait</span><span class="z-p">()</span> <span class="z-k">finally</span><span class="z-p">:</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">terminate</span><span class="z-p">()</span> <span class="z-c1"># ...</span> <span class="z-k">with</span> <span class="z-n">api</span><span class="z-o">.</span><span class="z-n">serve</span><span class="z-p">(</span><span class="z-n">port</span><span class="z-o">=</span><span class="z-n">port</span><span class="z-p">)</span> <span class="z-k">as</span> <span class="z-n">server_url</span><span class="z-p">:</span> <span class="z-n">logging</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s2">&#34;Server running at </span><span class="z-si">{</span><span class="z-n">server_url</span><span class="z-si">}</span><span class="z-s2">&#34;</span><span class="z-p">)</span> <span class="z-k">with</span> <span class="z-n">run_htmlpopup</span><span class="z-p">(</span><span class="z-n">server_url</span><span class="z-p">):</span> <span class="z-n">logging</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-s2">&#34;HTML popup started. Press Ctrl+C to stop.&#34;</span><span class="z-p">)</span></code></pre> <p>Check out <a href="https://github.com/abdusco/htmlpopup#readme" target="_blank" rel="noopener">JS API documentation</a> for more details. Download the latest release from the <a href="https://github.com/abdusco/htmlpopup/releases/latest" target="_blank" rel="noopener">Github releases page</a>.</p> Fri, 09 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/shimmer/ https://abdus.dev/projects/shimmer/ <div class="app-logo"><svg xmlns="http://www.w3.org/2000/svg" width="300" height="300" version="1.2" viewBox="0 0 1024 1024"><path d="M512 279c2 5 4 27 6 34a241 241 0 0 0 224 198c-75 8-143 42-186 105-27 40-37 79-44 125l-1-8a240 240 0 0 0-208-218l-25-4 34-5c39-7 76-23 107-47 42-33 72-78 85-130 4-17 6-32 8-50zM277 128l5 23c5 21 15 41 28 59 31 42 71 57 120 64-84 9-140 61-153 145l-1 4c-2-2-4-16-5-21a170 170 0 0 0-142-129 166 166 0 0 0 138-109c5-13 6-24 9-35zM749 601l5 23c19 74 71 111 142 126-64 8-117 51-139 112l-8 31c-9-43-20-74-54-103-31-26-58-35-97-40 86-12 135-65 151-149zM751 132l6 25c23 68 71 104 139 117-30 6-57 15-83 34-35 25-55 66-63 107l-1 6-1-1c-3-27-15-58-31-80-29-40-73-59-121-66 24-4 40-6 63-17 53-25 79-69 92-125zM275 601c2 1 6 25 7 29 21 72 74 108 146 120-31 2-66 16-91 35-35 26-55 65-61 107l-1 2-6-26a161 161 0 0 0-107-110l-33-8c82-14 135-65 146-149zM512 788l5 21c13 40 41 61 83 69-51 8-78 34-88 84-12-50-38-76-89-84 52-10 79-37 89-90zM878 426c2 3 3 15 5 20 4 11 10 22 17 31 16 19 38 32 63 34-26 8-42 12-60 33-16 17-20 30-25 52-12-52-41-76-92-85a98 98 0 0 0 91-84zM511 65c2 3 3 10 4 13 13 44 45 64 89 69-42 5-77 29-87 72l-5 22c-2-18-7-35-16-51-18-28-43-36-74-44 52-5 78-32 89-81zM146 427l4 14a101 101 0 0 0 82 70c-49 10-75 37-85 85l-5-16c-12-39-42-61-81-69 47-8 76-38 85-84z"/></svg></div> <style> .app-logo { width: 8rem; height: 8rem; margin-bottom: 2rem; } .app-logo svg { width: 100%; height: 100%; fill: var(--c-accent); } </style> <p>Shimmer is a live wallpaper app for Android that lets you apply cool GPU-accelerated blur and duotone effects to your wallpapers.</p> <a class="repo-card" href="https://github.com/abdusco/shimmer" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>shimmer</strong> </span> <span class="repo-card__description">A live wallpaper for Android that displays your photos with smooth GPU-based effects.</span> <span class="repo-card__meta"> <span>Kotlin</span> <span aria-label="9 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 9 </span> <span aria-label="1 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 1 </span> </span> </a> <p>I wanted to build this app ever since I first tried <a href="https://muzei.co/" target="_blank" rel="noopener">Muzei</a> back in 2014. I loved the idea of a wallpaper app just that gets out of your way. I didn't know much about OpenGL, but with the help of AI, I was able to create one with GPU accelerated blur effects (with clever optimizations). It's been a fun project to work on, and I've been fixing the bugs as I come across them.</p> <h2 id="features">Features</h2> <ul> <li>GPU accelerated gaussian blur and duotone effects.</li> <li>Chromatic aberration breathing effect on touch.</li> <li>Blur the wallpaper on lock, on timeout, or with a gesture.</li> <li>Many duotone color palettes to choose from, or you can pick your own colors.</li> <li>Cycle through your favorite wallpapers automatically on unlock or with a gesture.</li> <li>Marking wallpapers as favorites to easily find them later.</li> <li>Intent support for automation via Tasker or similar apps.</li> </ul> <h2 id="download">Download</h2> <p>Currently only on Github, source if also available if you wanna build it yourself. I am not planning to put it on the Play Store, but you can download the APK from the <a href="https://github.com/abdusco/shimmer/releases/latest" target="_blank" rel="noopener">releases page</a>.</p> Thu, 08 Oct 2026 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/prayertimes-v2/ https://abdus.dev/projects/prayertimes-v2/ <div class="prayertimes-logo"><svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24"> <path d="M0 0h24v24H0z" fill="none" /> <path fill="currentColor" d="M6.12 8h11.76m0 0c.07-.29.12-.59.12-.91c0-1.31-.65-2.53-1.74-3.25L12 1L7.74 3.84A3.89 3.89 0 0 0 6 7.09c0 .32.05.62.12.91" /> <path fill="currentColor" d="M24 7c0-1.1-2-3-2-3s-2 1.9-2 3c0 .74.4 1.38 1 1.72V13h-2V9H5v4H3V8.72c.6-.34 1-.98 1-1.72c0-1.1-2-3-2-3S0 5.9 0 7c0 .74.4 1.38 1 1.72V21h9v-6h4v6h9V8.72c.6-.34 1-.98 1-1.72" /> </svg> </div> <style> .prayertimes-logo { width: 8rem; height: 8rem; margin-bottom: 2rem; } .prayertimes-logo svg { width: 100%; height: 100%; color: var(--c-accent); } </style> <p>I built <a href="/projects/prayertimes/">the first version of this API</a> back in 2020, and it has been chugging along since then. However, it was based on scraping data from the <a href="https://namazvakitleri.diyanet.gov.tr/tr-TR" target="_blank" rel="noopener">Presidency of Religious Affairs of Turkey</a> website, which was not very stable, and I even have a cronjob that restarts the server because it kept breaking and I didn't have time to fix it.</p> <p>I started working on a newer version this year that functions by calculating the times based on the location and date. The calculations are based on studies published by Diyanet. This also turns out to be how Diyanet's own app works under the hood (I reverse engineered it). If you use the same coordinates as the app, you will get the exact same times. Slightly different coordinates will give slightly different times, but I think it's acceptable. It is also more stable and reliable than scraping the Diyanet website.</p> <p>I am using the city database from <a href="https://www.geonames.org/" target="_blank" rel="noopener">GeoNames</a> that contains <a href="https://download.geonames.org/export/dump/" target="_blank" rel="noopener">the most populous 5000 cities</a> to get the name + coordinates of a location, and then calculating the prayer times based on that.</p> <p>It also has <a href="https://prayertimes.api.abdus.dev/" target="_blank" rel="noopener">a web UI</a> that also works as a PWA, so you can install it on your phone. The API docs are available <a href="https://prayertimes.api.abdus.dev/api" target="_blank" rel="noopener">here</a>.</p> <a class="repo-card" href="https://git.abdus.dev/abdus/prayertimes-go" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdus</span> <span aria-hidden="true">/</span> <strong>prayertimes-go</strong> </span> <span class="repo-card__meta"> <span>Go</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> Sun, 17 Jul 2022 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/sniffing-io-reader-in-golang/ https://abdus.dev/posts/sniffing-io-reader-in-golang/ <h2 id="the-problem-reading-a-stream-multiple-times">The problem: reading a stream multiple times</h2> <p>I wanted to use <code>ffmpeg</code> to generate contact sheet for a video. This is the command I use to generate a thumbnail tile.</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell">cat /path/to/video.mp4 <span class="z-p">|</span> ffmpeg <span class="z-se">\ </span> -v error <span class="z-se">\ </span> -skip_frame nokey <span class="z-se">\ </span> <span class="z-c1"># use keyframes only (makes it fast)</span> -ss <span class="z-m">10</span> <span class="z-se">\ </span> <span class="z-c1"># skip the first 10 seconds</span> -i - <span class="z-se">\ </span> -vf <span class="z-s1">&#39;fps=1/60,scale=480:-1,tile=3x22&#39;</span> <span class="z-se">\ </span> <span class="z-c1"># 480px wide tiles in 3 columns every 60 seconds</span> -frames <span class="z-m">1</span> <span class="z-se">\ </span> -f image2pipe - <span class="z-se">\ </span> &gt; thumbs.jpg</code></pre> <p>The tricky part is to calculate <code>tile=3x22</code> argument. It's the grid (and the number of tiles) that determines how far into the video we're reading. If it's too small, the contact sheet summarize the whole video. If it's too many, the we get blank tiles at the end.</p> <p>There's a way out: if we know the duration of the video, we can calculate how many tiles there should be, so that we can cover the whole video.</p> <p>For that, I use <code>ffprobe</code>:</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell">cat /path/to/video.mp4 <span class="z-p">|</span> ffprobe <span class="z-se">\ </span> -v quiet <span class="z-se">\ </span> -i - <span class="z-se">\ </span> -print_format json <span class="z-se">\ </span> -show_entries <span class="z-nv">format</span><span class="z-o">=</span>duration</code></pre> <p>which returns the duration, and we can use that to calculate the size of the tile grid, for a given interval.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span> <span class="z-nt">&#34;format&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;duration&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;4059.051000&#34;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>I call <code>ffmpeg</code> from a CLI I've written in golang. I wanted to pipe the video stream to both <code>ffprobe</code> and <code>ffmpeg</code>, and reuse it without the caller knowing.</p> <pre data-lang="golang" data-highlighted="true" class="language-golang z-code z-chroma"><code class="language-golang" data-lang="golang"><span class="z-kd">func</span><span class="z-w"> </span><span class="z-nf">generateThumbs</span><span class="z-p">(</span><span class="z-nx">ctx</span><span class="z-w"> </span><span class="z-nx">context</span><span class="z-p">.</span><span class="z-nx">Context</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-w"> </span><span class="z-nx">io</span><span class="z-p">.</span><span class="z-nx">Reader</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">w</span><span class="z-w"> </span><span class="z-nx">io</span><span class="z-p">.</span><span class="z-nx">Writer</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-kt">error</span><span class="z-w"> </span><span class="z-p">{</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// ...</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">exec</span><span class="z-p">.</span><span class="z-nf">CommandContext</span><span class="z-p">(</span><span class="z-nx">ctx</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;ffprobe&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-cm">/*...*/</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-p">.</span><span class="z-nx">Stdin</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// parse duration</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// use the duration to calculate ffmpeg args</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">exec</span><span class="z-p">.</span><span class="z-nf">CommandContext</span><span class="z-p">(</span><span class="z-nx">ctx</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;ffmpeg&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-cm">/*...*/</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-p">.</span><span class="z-nx">Stdin</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-w"> </span><span class="z-c1">// reuse the video stream?</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-p">.</span><span class="z-nx">Stdout</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">w</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// ...</span><span class="z-w"> </span><span class="z-p">}</span><span class="z-w"> </span></code></pre> <p>Now we face a problem: whatever <code>ffprobe</code> reads from the video stream <code>r</code> is gone and <code>ffmpeg</code> cannot read it.</p> <p>We need to find a way to sniff the video properties and put what we consume back where we found it.</p> <h2 id="solution-io-teereader-and-io-multireader">Solution: <code>io.TeeReader</code> and <code>io.MultiReader</code></h2> <p>Luckily, <code>ffprobe</code> doesn't need to read the whole file to determine the video properties. It only reads a few MBs from the beginning. It doesn't hurt to keep a couple of MBs of video in memory.</p> <p>We can use <code>io.TeeReader</code> to keep those bits in a buffer. Then use a <code>io.MultiReader</code> to reconstruct the original stream by combining the buffer <code>buf</code> with the remainder of the input stream <code>r</code>. Then we pipe this to <code>ffmpeg</code>.</p> <pre data-lang="golang" data-highlighted="true" class="language-golang z-code z-chroma"><code class="language-golang" data-lang="golang"><span class="z-kd">var</span><span class="z-w"> </span><span class="z-nx">buf</span><span class="z-w"> </span><span class="z-nx">bytes</span><span class="z-p">.</span><span class="z-nx">Buffer</span><span class="z-w"> </span><span class="z-c1">// keep the bits ffprobe needs in a buffer</span><span class="z-w"> </span><span class="z-nx">tr</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">io</span><span class="z-p">.</span><span class="z-nf">TeeReader</span><span class="z-p">(</span><span class="z-nx">r</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-o">&amp;</span><span class="z-nx">buf</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">exec</span><span class="z-p">.</span><span class="z-nf">CommandContext</span><span class="z-p">(</span><span class="z-nx">ctx</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;ffprobe&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-cm">/*...*/</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-p">.</span><span class="z-nx">Stdin</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">tr</span><span class="z-w"> </span><span class="z-c1">// use the tee reader</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-c1">// ...</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">io</span><span class="z-p">.</span><span class="z-nf">MultiReader</span><span class="z-p">(</span><span class="z-o">&amp;</span><span class="z-nx">buf</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-c1">// then combine it back (in the right order)</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-w"> </span><span class="z-o">:=</span><span class="z-w"> </span><span class="z-nx">exec</span><span class="z-p">.</span><span class="z-nf">CommandContext</span><span class="z-p">(</span><span class="z-nx">ctx</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-s">&#34;ffmpeg&#34;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-cm">/*...*/</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-nx">cmd</span><span class="z-p">.</span><span class="z-nx">Stdin</span><span class="z-w"> </span><span class="z-p">=</span><span class="z-w"> </span><span class="z-nx">r</span><span class="z-w"> </span><span class="z-c1">// re-constructed input</span><span class="z-w"> </span></code></pre> <p>Now the contraption works great. <code>ffprobe</code> reads a couple megs, finds the duration, then ffmpeg reads the whole file again and generates the thumbs.</p> Thu, 21 Oct 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/django-orm-standalone/ https://abdus.dev/posts/django-orm-standalone/ <p>Django's ORM is really simple to use and migrations support is great. When I modify a model and run <code>makemigrations</code> it generates a migration from the changes, and I can apply it with <code>migrate</code> and I'm done.</p> <p>I wanted to use it in a serverless function to write in a remote Postgres instance. Django doesn't have a separate package for it's ORM, but it's still possible to use it on its own.</p> <h2 id="minimal-boilerplate">Minimal boilerplate</h2> <p>First, I've created a <code>manage.py</code> to configure Django. It also runs the management CLI. I only need to specify the <code>INSTALLED_APPS</code> for model discovery to work, and a database connection.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-ch">#!/usr/bin/env python</span> <span class="z-k">def</span> <span class="z-nf">init_django</span><span class="z-p">():</span> <span class="z-kn">import</span> <span class="z-nn">django</span> <span class="z-kn">from</span> <span class="z-nn">django.conf</span> <span class="z-kn">import</span> <span class="z-n">settings</span> <span class="z-k">if</span> <span class="z-n">settings</span><span class="z-o">.</span><span class="z-n">configured</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-n">settings</span><span class="z-o">.</span><span class="z-n">configure</span><span class="z-p">(</span> <span class="z-n">INSTALLED_APPS</span><span class="z-o">=</span><span class="z-p">[</span> <span class="z-s1">&#39;db&#39;</span><span class="z-p">,</span> <span class="z-p">],</span> <span class="z-n">DATABASES</span><span class="z-o">=</span><span class="z-p">{</span> <span class="z-s1">&#39;default&#39;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-s1">&#39;ENGINE&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;django.db.backends.postgresql&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;NAME&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;myapp&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;USER&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;myapp_user&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;PASSWORD&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;myapp&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;HOST&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;127.0.0.1&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;PORT&#39;</span><span class="z-p">:</span> <span class="z-s1">&#39;5432&#39;</span><span class="z-p">,</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">)</span> <span class="z-n">django</span><span class="z-o">.</span><span class="z-n">setup</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s2">&#34;__main__&#34;</span><span class="z-p">:</span> <span class="z-kn">from</span> <span class="z-nn">django.core.management</span> <span class="z-kn">import</span> <span class="z-n">execute_from_command_line</span> <span class="z-n">init_django</span><span class="z-p">()</span> <span class="z-n">execute_from_command_line</span><span class="z-p">()</span></code></pre> <p>I've created a module called <code>db</code> to act as a Django app and placed a <code>models.py</code> in it.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-c1"># db/models.py</span> <span class="z-kn">from</span> <span class="z-nn">django.db</span> <span class="z-kn">import</span> <span class="z-n">models</span> <span class="z-kn">from</span> <span class="z-nn">manage</span> <span class="z-kn">import</span> <span class="z-n">init_django</span> <span class="z-n">init_django</span><span class="z-p">()</span> <span class="z-k">class</span> <span class="z-nc">Model</span><span class="z-p">(</span><span class="z-n">models</span><span class="z-o">.</span><span class="z-n">Model</span><span class="z-p">):</span> <span class="z-nb">id</span> <span class="z-o">=</span> <span class="z-n">models</span><span class="z-o">.</span><span class="z-n">AutoField</span><span class="z-p">(</span><span class="z-n">primary_key</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-n">created_at</span> <span class="z-o">=</span> <span class="z-n">models</span><span class="z-o">.</span><span class="z-n">DateTimeField</span><span class="z-p">(</span><span class="z-n">auto_now_add</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">db_index</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-n">updated_at</span> <span class="z-o">=</span> <span class="z-n">models</span><span class="z-o">.</span><span class="z-n">DateTimeField</span><span class="z-p">(</span><span class="z-n">auto_now</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-k">class</span> <span class="z-nc">Meta</span><span class="z-p">:</span> <span class="z-n">abstract</span> <span class="z-o">=</span> <span class="z-kc">True</span> <span class="z-c1"># define models here</span> <span class="z-k">class</span> <span class="z-nc">Post</span><span class="z-p">(</span><span class="z-n">Model</span><span class="z-p">):</span> <span class="z-o">...</span></code></pre> <p>Django needs to initialized for models to work, that's why I call <code>init_django()</code> to initialize it before defining the models.</p> <p>Here's how the final folder structure looks like:</p> <pre><code>. |-- db | |-- __init__.py | `-- models.py |-- manage.py |-- requirements.txt </code></pre> <p>That's about it. Now I can build models and update the database schema:</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell"><span class="z-c1"># create a migration under `./db/migrations`</span> $ python manage.py makemigrations db <span class="z-c1"># apply migrations</span> $ python manage.py migrate</code></pre> <p>To use the models, I import them directly like any other class.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">from</span> <span class="z-nn">db.models</span> <span class="z-kn">import</span> <span class="z-n">Post</span> <span class="z-k">for</span> <span class="z-n">it</span> <span class="z-ow">in</span> <span class="z-n">Post</span><span class="z-o">.</span><span class="z-n">objects</span><span class="z-o">.</span><span class="z-n">all</span><span class="z-p">():</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-n">it</span><span class="z-p">)</span></code></pre> Thu, 29 Jul 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/aspnetcore-multiple-sessions/ https://abdus.dev/posts/aspnetcore-multiple-sessions/ <p>Assume we have an app that gives users the option to <a href="https://docs.microsoft.com/en-us/aspnet/core/security/authentication/social/" target="_blank" rel="noopener">login with Google, GitHub, Facebook, etc.</a>. We want to let users login with any identity provider and save their session to access the claims they provide later.</p> <p>For example, if I login with Google, it returns claims about the logged-in user, such as his name, email, account id. The same holds true for GitHub and all other OIDC providers.</p> <h2 id="problem">Problem</h2> <p>If the user signs in with his Google account, then signs in with GitHub, the session cookie for Google is overwritten by that of GitHub's.</p> <p>How can we access all active sessions and read the claims?</p> <h2 id="setting-up-authentication-for-google-and-github">Setting up authentication for Google and GitHub</h2> <p>We define the authentication schemes like this:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddAuthentication</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">DefaultScheme</span> <span class="z-p">=</span> <span class="z-n">CookieAuthenticationDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">AddCookie</span><span class="z-p">()</span> <span class="z-p">.</span><span class="z-n">AddGoogle</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-c1">// set the client id + secret</span> <span class="z-n">Configuration</span><span class="z-p">.</span><span class="z-n">GetSection</span><span class="z-p">(</span><span class="z-s">&#34;Google&#34;</span><span class="z-p">).</span><span class="z-n">Bind</span><span class="z-p">(</span><span class="z-n">options</span><span class="z-p">);</span> <span class="z-p">})</span> <span class="z-p">.</span><span class="z-n">AddGitHub</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-c1">// set the client id + secret</span> <span class="z-n">Configuration</span><span class="z-p">.</span><span class="z-n">GetSection</span><span class="z-p">(</span><span class="z-s">&#34;GitHub&#34;</span><span class="z-p">).</span><span class="z-n">Bind</span><span class="z-p">(</span><span class="z-n">options</span><span class="z-p">);</span> <span class="z-p">});</span> <span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddAuthorization</span><span class="z-p">();</span></code></pre> <p>We use a cookie to hold the user session, and a couple of external id providers to offer the users. The credentials are stored in <code>appsettings.json</code> in this format, so they play nice<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> with the <code>IConfiguration.Bind()</code> method:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span> <span class="z-nt">&#34;GitHub&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;ClientId&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;c27966ecf4087378&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;ClientSecret&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;66c71cdac566e43d869dd3c70318ed5e8df22af7a565f068adef6a6bacf6aca7&#34;</span> <span class="z-p">},</span> <span class="z-nt">&#34;Google&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;ClientId&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;ed5a20d944fd12ebe33ad0de.apps.googleusercontent.com&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;ClientSecret&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;87a3a35d003d1622c6bdbf454c0e3608ca68b1e780c965df4c52dc2f9b2c9d78&#34;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>Now when we <a href="https://docs.microsoft.com/en-us/aspnet/core/security/authentication/?view=aspnetcore-5.0#challenge" target="_blank" rel="noopener">issue a challenge</a>, the user is forced to login with a provider:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[ApiController]</span> <span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">AccountController</span> <span class="z-p">:</span> <span class="z-n">ControllerBase</span> <span class="z-p">{</span> <span class="z-na"> [AllowAnonymous]</span> <span class="z-na"> [HttpGet(&#34;/login-google&#34;)]</span> <span class="z-kd">public</span> <span class="z-n">ActionResult</span> <span class="z-n">LoginGoogle</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-n">Challenge</span><span class="z-p">(</span> <span class="z-k">new</span> <span class="z-n">AuthenticationProperties</span> <span class="z-p">{</span> <span class="z-n">RedirectUri</span> <span class="z-p">=</span> <span class="z-n">Url</span><span class="z-p">.</span><span class="z-n">Action</span><span class="z-p">(</span><span class="z-s">&#34;WhoAmI&#34;</span><span class="z-p">),</span> <span class="z-p">},</span> <span class="z-n">GoogleDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span> <span class="z-p">);</span> <span class="z-p">}</span> <span class="z-na"> </span><span class="z-na"> [AllowAnonymous]</span> <span class="z-na"> [HttpGet(&#34;/login-GitHub&#34;)]</span> <span class="z-kd">public</span> <span class="z-n">ActionResult</span> <span class="z-n">LoginGitHub</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-n">Challenge</span><span class="z-p">(</span> <span class="z-k">new</span> <span class="z-n">AuthenticationProperties</span> <span class="z-p">{</span> <span class="z-n">RedirectUri</span> <span class="z-p">=</span> <span class="z-n">Url</span><span class="z-p">.</span><span class="z-n">Action</span><span class="z-p">(</span><span class="z-s">&#34;WhoAmI&#34;</span><span class="z-p">),</span> <span class="z-p">},</span> <span class="z-n">GitHubAuthenticationDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span> <span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>Visiting <code>/login-google</code>, the app redirects us to Google sign in page, and after logging in, we are redirected back to <code>WhoAmI</code> action, which dumps all user claims.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[AllowAnonymous]</span> <span class="z-na">[HttpGet(&#34;/me&#34;)]</span> <span class="z-kd">public</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&gt;</span> <span class="z-n">WhoAmI</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-c1">// dump all claims</span> <span class="z-k">return</span> <span class="z-n">Ok</span><span class="z-p">(</span> <span class="z-n">User</span><span class="z-p">.</span><span class="z-n">Identities</span><span class="z-p">.</span><span class="z-n">Select</span><span class="z-p">(</span> <span class="z-n">id</span> <span class="z-p">=&gt;</span> <span class="z-k">new</span> <span class="z-p">{</span> <span class="z-n">id</span><span class="z-p">.</span><span class="z-n">AuthenticationType</span><span class="z-p">,</span> <span class="z-n">Claims</span> <span class="z-p">=</span> <span class="z-n">id</span><span class="z-p">.</span><span class="z-n">Claims</span><span class="z-p">.</span><span class="z-n">Select</span><span class="z-p">(</span><span class="z-n">c</span> <span class="z-p">=&gt;</span> <span class="z-k">new</span> <span class="z-p">{</span> <span class="z-n">c</span><span class="z-p">.</span><span class="z-n">Type</span><span class="z-p">,</span> <span class="z-n">c</span><span class="z-p">.</span><span class="z-n">Value</span> <span class="z-p">})</span> <span class="z-p">}</span> <span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">ToList</span><span class="z-p">()</span> <span class="z-p">);</span> <span class="z-p">}</span></code></pre> <p>The result is an empty JSON:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;authenticationType&#34;</span><span class="z-p">:</span> <span class="z-kc">null</span><span class="z-p">,</span> <span class="z-nt">&#34;claims&#34;</span><span class="z-p">:</span> <span class="z-p">[]</span> <span class="z-p">}</span> <span class="z-p">]</span></code></pre> <p><code>WhoAmI</code> action doesn't do anything, because we're bypassing authentication with <code>[AllowAnonymous]</code>. We need to add <code>[Authorize]</code> attribute and specify an authentication scheme to access the claims for that session.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[Authorize(AuthenticationSchemes = GoogleDefaults.AuthenticationScheme)]</span> <span class="z-na">[HttpGet(&#34;/me&#34;)]</span> <span class="z-kd">public</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&gt;</span> <span class="z-n">WhoAmI</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-c1">// `User` now contains Google claims</span> <span class="z-c1">// ...</span> <span class="z-p">}</span></code></pre> <h2 id="disappearing-sessions">Disappearing sessions</h2> <p>Now if we log in with GitHub by visiting <code>/login-github</code>, and sign in with our credentials, we're redirected to <code>/me</code>, but it keeps redirecting us to Google's login page. What is happening here?</p> <p>When we've signed in on GitHub, the session is saved to the cookie we've added with <code>.AddCookie()</code>, since it is the default authentication scheme. And unless specified otherwise, all authentication actions default to the <code>DefaultScheme</code> .</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddAuthentication</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">DefaultScheme</span> <span class="z-p">=</span> <span class="z-n">CookieAuthenticationDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">AddCookie</span><span class="z-p">()</span> <span class="z-c1">// ...</span></code></pre> <p>To prevent losing our Google session, we need a separate cookie to store it. Let's add another one and give both a better name while we're at it. Then we use those cookies to store Google &amp; GitHub sessions:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddAuthentication</span><span class="z-p">()</span> <span class="z-p">.</span><span class="z-n">AddCookie</span><span class="z-p">(</span><span class="z-s">&#34;GoogleSession&#34;</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">AddCookie</span><span class="z-p">(</span><span class="z-s">&#34;GitHubSession&#34;</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">AddGoogle</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-c1">// save session to this cookie</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">SignInScheme</span> <span class="z-p">=</span> <span class="z-s">&#34;GoogleSession&#34;</span><span class="z-p">;</span> <span class="z-p">})</span> <span class="z-p">.</span><span class="z-n">AddGitHub</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-c1">// save session to this cookie</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">SignInScheme</span> <span class="z-p">=</span> <span class="z-s">&#34;GitHubSession&#34;</span><span class="z-p">;</span> <span class="z-p">});</span></code></pre> <p>Now <code>WhoAmI</code> action works as expected and we get a list of Google claims thanks to <code>[Authorize]</code> attribute:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;authenticationType&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Google&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;claims&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;type&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;value&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;13131323423131413123&#34;</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;type&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;value&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Abdussamet Koçak&#34;</span> <span class="z-p">},</span> <span class="z-c1">// ... </span> <span class="z-p">]</span> <span class="z-p">}</span> <span class="z-p">]</span></code></pre> <p>If we change the <code>[Authorize]</code> attribute to <code>GitHub</code>, we get GitHub claims. But not both. For that we need to specify the names of both separated with a comma:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[Authorize(AuthenticationSchemes = &#34;Google,GitHub&#34;)]</span></code></pre> <p>We can't concatenate the constants <code>GoogleDefaults.AuthenticationScheme</code> and <code>GitHubAuthenticationDefaults.AuthenticationScheme</code> here, because C# doesn't support concatenated constants (yet, but <a href="https://GitHub.com/dotnet/csharplang/issues/2951" target="_blank" rel="noopener">it's on its way to C# 10</a>), so instead, we're using their values.</p> <p>Now <code>/me</code> returns claims for both sessions:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;authenticationType&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;GitHub&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;claims&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-c1">// ... GitHub claims </span> <span class="z-p">]</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;authenticationType&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Google&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;claims&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-c1">// ... google claims </span> <span class="z-p">]</span> <span class="z-p">}</span> <span class="z-p">]</span></code></pre> <h2 id="setting-a-default-authorization-policy">Setting a default authorization policy</h2> <p>Specifying all these magic values by hand is tedious. Luckily ASP.NET Core gives us some other options to achieve the same result.</p> <p>After configuring a default authorization policy that forces the user authenticate with multiple schemes,</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddAuthorization</span><span class="z-p">(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">DefaultPolicy</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">AuthorizationPolicyBuilder</span><span class="z-p">(</span> <span class="z-n">GoogleDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span><span class="z-p">,</span> <span class="z-n">GitHubAuthenticationDefaults</span><span class="z-p">.</span><span class="z-n">AuthenticationScheme</span> <span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">RequireAuthenticatedUser</span><span class="z-p">()</span> <span class="z-p">.</span><span class="z-n">Build</span><span class="z-p">()</span> <span class="z-p">);</span></code></pre> <p>we can get rid of auth the arguments in <code>[Authorize]</code> and use the default policy.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[Authorize]</span> <span class="z-na">[HttpGet(&#34;/me&#34;)]</span> <span class="z-kd">public</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&gt;</span> <span class="z-n">WhoAmI</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-p">...</span> <span class="z-p">}</span></code></pre> <h2 id="resources">Resources</h2> <ul> <li><a href="https://docs.microsoft.com/en-us/aspnet/core/security/authentication/" target="_blank" rel="noopener">Microsoft's docs on getting started with ASP.NET Core auth</a></li> <li><a href="https://docs.microsoft.com/en-us/aspnet/core/security/authentication/social/" target="_blank" rel="noopener">Microsoft's docs on setting up social authentication</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>Keys in the <code>appsettings.json</code> configuration file correspond to the properties in the option class of their respective auth schemes.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Tue, 29 Jun 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/aspnetcore-inject-cancellationtoken-with-timeout/ https://abdus.dev/posts/aspnetcore-inject-cancellationtoken-with-timeout/ <p>ASP.NET Core allows us to inject a <code>CancellationToken</code> in actions:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[HttpGet]</span> <span class="z-kd">public</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&gt;</span> <span class="z-n">GetLoggedInUser</span><span class="z-p">(</span><span class="z-n">CancellationToken</span> <span class="z-n">cancellationToken</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-p">...</span> <span class="z-p">}</span></code></pre> <p>This cancellation token is actually bound to <code>HttpContext.RequestAborted</code>, and it is signalled when the request is cancelled.</p> <p>We can flow this token down to any async method that accepts it and help coordinate their cancellation if the request is cancelled by the user (due to page navigation, for example). This prevents wasting time doing heavy work, and having to throw it all away for a client that's already gone.</p> <h2 id="problem">Problem</h2> <p>How can we add a default timeout to this token, so that the operation is cancelled either when the request is cancelled or it takes too long?</p> <h2 id="model-binding">Model binding</h2> <div class="has-recommend"> <p>MVC framework performs model binding on action parameters by parsing the request into usable data structures. It is used for reading the request body serialized as JSON to a class, or extracting a form field, or reading file stream into an <code>IFormFile</code>, among other things.</p> <div class="recommend"> <p><a href="/posts/aspnetcore-model-binding-json-query-params/">Model binding for query parameters encoded as JSON in ASP.NET Core</a></p> </div> </div> <p><code>CancellationToken</code>s are bound to <code>HttpContext.RequestAborted</code> by <code>CancellationTokenModelBinder</code> class. We'll replace it with our own implementation that will set the token to expire with a timeout.</p> <p>But first, how do we join cancellation tokens together?</p> <h2 id="combining-cancellation-tokens">Combining cancellation tokens</h2> <p>We can merge multiple cancellation tokens into one with <code>CancellationTokenSource.CreateLinkedTokenSource</code> method. Here we create a timeout token, and link it to another:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kt">var</span> <span class="z-n">cancellationToken</span> <span class="z-p">=</span> <span class="z-c1">// ...</span> <span class="z-kt">var</span> <span class="z-n">timeoutCts</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">CancellationTokenSource</span><span class="z-p">();</span> <span class="z-n">timeoutCts</span><span class="z-p">.</span><span class="z-n">CancelAfter</span><span class="z-p">(</span><span class="z-n">TimeSpan</span><span class="z-p">.</span><span class="z-n">FromSeconds</span><span class="z-p">(</span><span class="z-m">10</span><span class="z-p">));</span> <span class="z-kt">var</span> <span class="z-n">combinedCts</span> <span class="z-p">=</span> <span class="z-n">CancellationTokenSource</span><span class="z-p">.</span><span class="z-n">CreateLinkedTokenSource</span><span class="z-p">(</span><span class="z-n">timeoutCts</span><span class="z-p">.</span><span class="z-n">Token</span><span class="z-p">,</span> <span class="z-n">cancellationToken</span><span class="z-p">);</span></code></pre> <p>Time to put it to use.</p> <h2 id="building-a-model-binder-for-cancellation-tokens">Building a model binder for cancellation tokens</h2> <p>Looking at the source code for <code>CancellationTokenModelBinder</code>, it doesn't do much, but we'll still use it as our starting point. We'll subclass it, but with a twist. Because it doesn't mark <code>BindModelAsync</code> method as <code>virtual</code> we can't override it with an <code>override</code> keyword. By implementing the <code>IModelBinder</code> interface that the base class implements, <a href="/posts/csharp-overriding-non-virtual-methods/">we can get over this limitation</a> and still &quot;override&quot; it without actually using <code>override</code>.</p> <p>After letting the base class perform the binding, we'll take the bound token, and replace it with a new one.</p> <p>Since we've replaced the token object, we'll have to rebuild the validation state. In fact, why not just copy over what ASP.NET Core team has put there because I don't know enough about to disagree with them.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">private</span> <span class="z-k">class</span> <span class="z-nc">TimeoutCancellationTokenModelBinder</span> <span class="z-p">:</span> <span class="z-n">CancellationTokenModelBinder</span><span class="z-p">,</span> <span class="z-n">IModelBinder</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-k">new</span> <span class="z-kd">async</span> <span class="z-n">Task</span> <span class="z-n">BindModelAsync</span><span class="z-p">(</span><span class="z-n">ModelBindingContext</span> <span class="z-n">bindingContext</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">await</span> <span class="z-k">base</span><span class="z-p">.</span><span class="z-n">BindModelAsync</span><span class="z-p">(</span><span class="z-n">bindingContext</span><span class="z-p">);</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">Result</span><span class="z-p">.</span><span class="z-n">Model</span> <span class="z-k">is</span> <span class="z-n">CancellationToken</span> <span class="z-n">cancellationToken</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// combine the default token with a timeout</span> <span class="z-kt">var</span> <span class="z-n">timeoutCts</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">CancellationTokenSource</span><span class="z-p">();</span> <span class="z-n">timeoutCts</span><span class="z-p">.</span><span class="z-n">CancelAfter</span><span class="z-p">(</span><span class="z-n">TimeSpan</span><span class="z-p">.</span><span class="z-n">FromSeconds</span><span class="z-p">(</span><span class="z-m">5</span><span class="z-p">));</span> <span class="z-kt">var</span> <span class="z-n">combinedCts</span> <span class="z-p">=</span> <span class="z-n">CancellationTokenSource</span><span class="z-p">.</span><span class="z-n">CreateLinkedTokenSource</span><span class="z-p">(</span><span class="z-n">timeoutCts</span><span class="z-p">.</span><span class="z-n">Token</span><span class="z-p">,</span> <span class="z-n">cancellationToken</span><span class="z-p">);</span> <span class="z-c1">// We need to force boxing now, so we can insert the same reference to the boxed CancellationToken</span> <span class="z-c1">// in both the ValidationState and ModelBindingResult.</span> <span class="z-c1">//</span> <span class="z-c1">// DO NOT simplify this code by removing the cast.</span> <span class="z-kt">var</span> <span class="z-n">model</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-kt">object</span><span class="z-p">)</span><span class="z-n">combinedCts</span><span class="z-p">.</span><span class="z-n">Token</span><span class="z-p">;</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ValidationState</span><span class="z-p">.</span><span class="z-n">Clear</span><span class="z-p">();</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ValidationState</span><span class="z-p">.</span><span class="z-n">Add</span><span class="z-p">(</span><span class="z-n">model</span><span class="z-p">,</span> <span class="z-k">new</span> <span class="z-n">ValidationStateEntry</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-n">SuppressValidation</span> <span class="z-p">=</span> <span class="z-kc">true</span> <span class="z-p">});</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">Result</span> <span class="z-p">=</span> <span class="z-n">ModelBindingResult</span><span class="z-p">.</span><span class="z-n">Success</span><span class="z-p">(</span><span class="z-n">model</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>We also need a provider to introduce this model binder to MVC pipeline:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">TimeoutCancellationTokenModelBinderProvider</span> <span class="z-p">:</span> <span class="z-n">IModelBinderProvider</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-n">IModelBinder</span><span class="z-p">?</span> <span class="z-n">GetBinder</span><span class="z-p">(</span><span class="z-n">ModelBinderProviderContext</span> <span class="z-n">context</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-n">context</span><span class="z-p">?.</span><span class="z-n">Metadata</span><span class="z-p">.</span><span class="z-n">ModelType</span> <span class="z-p">!=</span> <span class="z-k">typeof</span><span class="z-p">(</span><span class="z-n">CancellationToken</span><span class="z-p">))</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-kc">null</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-k">new</span> <span class="z-n">TimeoutCancellationTokenModelBinder</span><span class="z-p">();</span> <span class="z-p">}</span> <span class="z-kd">private</span> <span class="z-k">class</span> <span class="z-nc">TimeoutCancellationTokenModelBinder</span> <span class="z-p">:</span> <span class="z-n">CancellationTokenModelBinder</span><span class="z-p">,</span> <span class="z-n">IModelBinder</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <h2 id="replacing-model-binders">Replacing model binders</h2> <p>MVC has <a href="https://github.com/dotnet/aspnetcore/blob/9fa3421e1302bce7dbc50269edbd3072907e7832/src/Mvc/Mvc.Core/src/Infrastructure/MvcCoreMvcOptionsSetup.cs#L62-L78" target="_blank" rel="noopener">many model binders in place</a> to handle all kinds of parsing and binding for various types of parameters and sources. We need to put ours in the first place to run it before all others. We'll also remove the original cancellation token binder in order to prevent both binders from working over themselves.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">Configure</span><span class="z-p">&lt;</span><span class="z-n">MvcOptions</span><span class="z-p">&gt;(</span><span class="z-n">options</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">ModelBinderProviders</span><span class="z-p">.</span><span class="z-n">RemoveType</span><span class="z-p">&lt;</span><span class="z-n">CancellationTokenModelBinderProvider</span><span class="z-p">&gt;();</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">ModelBinderProviders</span><span class="z-p">.</span><span class="z-n">Insert</span><span class="z-p">(</span><span class="z-m">0</span><span class="z-p">,</span> <span class="z-k">new</span> <span class="z-n">TimeoutCancellationTokenModelBinderProvider</span><span class="z-p">());</span> <span class="z-p">});</span></code></pre> <h2 id="usage">Usage</h2> <p>Let's put it to practice. We'll try to run a long async task that takes 6 seconds. The <code>CancellationToken</code> parameter will be bound by our binder and it is set to expire after 5 seconds. After 5 seconds, it will throw an <code>TaskCanceledException</code> and cancel the execution.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[HttpGet(&#34;&#34;)]</span> <span class="z-kd">public</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">IActionResult</span><span class="z-p">&gt;</span> <span class="z-n">Index</span><span class="z-p">(</span><span class="z-n">CancellationToken</span> <span class="z-n">cancellationToken</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">await</span> <span class="z-n">Task</span><span class="z-p">.</span><span class="z-n">Delay</span><span class="z-p">(</span><span class="z-n">TimeSpan</span><span class="z-p">.</span><span class="z-n">FromSeconds</span><span class="z-p">(</span><span class="z-m">6</span><span class="z-p">),</span> <span class="z-n">cancellationToken</span><span class="z-p">);</span> <span class="z-k">return</span> <span class="z-n">Ok</span><span class="z-p">(</span><span class="z-s">&#34;hey&#34;</span><span class="z-p">);</span> <span class="z-p">}</span></code></pre> <p>Now let's get rid of the hardcoded values.</p> <h2 id="making-it-configurable">Making it configurable</h2> <p>We've set the timeout to 5 seconds, but it'd be better if it were configurable. We'll use the <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/options" target="_blank" rel="noopener">options pattern</a> for this. Let's create a class to encapsulate the options.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">TimeoutOptions</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-kt">int</span> <span class="z-n">TimeoutSeconds</span> <span class="z-p">{</span> <span class="z-k">get</span><span class="z-p">;</span> <span class="z-k">set</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">=</span> <span class="z-m">10</span><span class="z-p">;</span> <span class="z-c1">// seconds</span> <span class="z-kd">public</span> <span class="z-n">TimeSpan</span> <span class="z-n">Timeout</span> <span class="z-p">=&gt;</span> <span class="z-n">TimeSpan</span><span class="z-p">.</span><span class="z-n">FromSeconds</span><span class="z-p">(</span><span class="z-n">TimeoutSeconds</span><span class="z-p">);</span> <span class="z-p">}</span></code></pre> <p>We can configure the timeout in <code>ConfigureServices</code>:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-n">services</span><span class="z-p">.</span><span class="z-n">Configure</span><span class="z-p">&lt;</span><span class="z-n">TimeoutOptions</span><span class="z-p">&gt;(</span><span class="z-n">configuration</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-n">configuration</span><span class="z-p">.</span><span class="z-n">TimeoutSeconds</span> <span class="z-p">=</span> <span class="z-m">10</span><span class="z-p">;</span> <span class="z-p">});</span></code></pre> <p>and take this as a parameter in the binder provider.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">TimeoutCancellationTokenModelBinderProvider</span> <span class="z-p">:</span> <span class="z-n">IModelBinderProvider</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-n">IModelBinder</span><span class="z-p">?</span> <span class="z-n">GetBinder</span><span class="z-p">(</span><span class="z-n">ModelBinderProviderContext</span> <span class="z-n">context</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-c1">// resolve the configuration from the container</span> <span class="z-kt">var</span> <span class="z-n">config</span> <span class="z-p">=</span> <span class="z-n">context</span><span class="z-p">.</span><span class="z-n">Services</span><span class="z-p">.</span><span class="z-n">GetRequiredService</span><span class="z-p">&lt;</span><span class="z-n">IOptions</span><span class="z-p">&lt;</span><span class="z-n">TimeoutOptions</span><span class="z-p">&gt;&gt;().</span><span class="z-n">Value</span><span class="z-p">;</span> <span class="z-c1">// pass it down to the binder</span> <span class="z-k">return</span> <span class="z-k">new</span> <span class="z-n">TimeoutCancellationTokenModelBinder</span><span class="z-p">(</span><span class="z-n">config</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-kd">private</span> <span class="z-k">class</span> <span class="z-nc">TimeoutCancellationTokenModelBinder</span> <span class="z-p">:</span> <span class="z-n">CancellationTokenModelBinder</span><span class="z-p">,</span> <span class="z-n">IModelBinder</span> <span class="z-p">{</span> <span class="z-c1">// then inject it to the binder</span> <span class="z-kd">private</span> <span class="z-k">readonly</span> <span class="z-n">TimeoutOptions</span> <span class="z-n">_options</span><span class="z-p">;</span> <span class="z-kd">public</span> <span class="z-n">TimeoutCancellationTokenModelBinder</span><span class="z-p">(</span><span class="z-n">TimeoutOptions</span> <span class="z-n">options</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_options</span> <span class="z-p">=</span> <span class="z-n">options</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-k">new</span> <span class="z-kd">async</span> <span class="z-n">Task</span> <span class="z-n">BindModelAsync</span><span class="z-p">(</span><span class="z-n">ModelBindingContext</span> <span class="z-n">bindingContext</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-cm">/* ... */</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// use the configured timeout value</span> <span class="z-kt">var</span> <span class="z-n">timeoutCts</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">CancellationTokenSource</span><span class="z-p">();</span> <span class="z-n">timeoutCts</span><span class="z-p">.</span><span class="z-n">CancelAfter</span><span class="z-p">(</span><span class="z-n">_options</span><span class="z-p">.</span><span class="z-n">Timeout</span><span class="z-p">);</span> <span class="z-c1">// ...</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>That's it. Thanks for reading.</p> Mon, 28 Jun 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/csharp-overriding-non-virtual-methods/ https://abdus.dev/posts/csharp-overriding-non-virtual-methods/ <p>C# lets us add <code>virtual</code> modifier to a method to indicate that it can be overridden by the subclasses of that (non-<code>sealed</code>) class.</p> <h2 id="problem">Problem</h2> <p><a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/override" target="_blank" rel="noopener">C# spec says</a>:</p> <blockquote> <p>You cannot override a non-virtual or static method. The overridden base method must be <code>virtual</code>, <code>abstract</code>, or <code>override</code>.</p> </blockquote> <p>When working with a library, sometimes we <em>really</em> need to override a method to plug our own logic, but darn it, it's not <code>virtual</code>, definitely not <code>abstract</code>, it doesn't have <code>override</code> either. What do we do?</p> <h2 id="trick">Trick</h2> <p>We have one crucial condition for this trick to work: <strong>the method we want to override must come from an interface implemented by the class.</strong></p> <p>Take this example:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-k">interface</span> <span class="z-nc">IEmailSender</span> <span class="z-p">{</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">SendResult</span><span class="z-p">&gt;</span> <span class="z-n">SendAsync</span><span class="z-p">(</span><span class="z-n">MailMessage</span> <span class="z-n">email</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">DefaultEmailSender</span> <span class="z-p">:</span> <span class="z-n">IEmailSender</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">SendResult</span><span class="z-p">&gt;</span> <span class="z-n">SendAsync</span><span class="z-p">(</span><span class="z-n">MailMessage</span> <span class="z-n">email</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>We want to override the <code>SendAsync</code> method, but it's not <code>virtual</code>/<code>override</code>. Don't fret, in three easy steps, we can override it:</p> <ol start="0"> <li>Create a subclass</li> <li>Implement the interface that provides the method</li> <li>Add <code>new</code> modifier</li> </ol> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">ConsoleEmailSender</span> <span class="z-p">:</span> <span class="z-n">DefaultEmailSender</span><span class="z-p">,</span> <span class="z-n">IEmailSender</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-k">new</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">SendResult</span><span class="z-p">&gt;</span> <span class="z-n">SendAsync</span><span class="z-p">(</span><span class="z-n">MailMessage</span> <span class="z-n">email</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// custom logic!</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>We can modify the arguments, call the base implementation and modify the result:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">ConsoleEmailSender</span> <span class="z-p">:</span> <span class="z-n">DefaultEmailSender</span><span class="z-p">,</span> <span class="z-n">IEmailSender</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-k">new</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">SendResult</span><span class="z-p">&gt;</span> <span class="z-n">SendAsync</span><span class="z-p">(</span><span class="z-n">MailMessage</span> <span class="z-n">email</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ... modify the arguments</span> <span class="z-n">Console</span><span class="z-p">.</span><span class="z-n">WriteLine</span><span class="z-p">(</span><span class="z-n">email</span><span class="z-p">.</span><span class="z-n">Subject</span><span class="z-p">);</span> <span class="z-n">email</span><span class="z-p">.</span><span class="z-n">Subject</span> <span class="z-p">=</span> <span class="z-s">&#34;console&#34;</span><span class="z-p">;</span> <span class="z-kt">var</span> <span class="z-n">result</span> <span class="z-p">=</span> <span class="z-k">await</span> <span class="z-k">base</span><span class="z-p">.</span><span class="z-n">SendAsync</span><span class="z-p">(</span><span class="z-n">bindingContext</span><span class="z-p">);</span> <span class="z-c1">// ... modify the results</span> <span class="z-k">return</span> <span class="z-n">result</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>It is a neat little trick that I resort to from time to time.</p> Tue, 30 Mar 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/aspnetcore-model-binding-json-query-params/ https://abdus.dev/posts/aspnetcore-model-binding-json-query-params/ <p>I've been building and an API that supports seaching with complex filters. Think of product filtering you see on most shopping websites, with inputs for price, rating, brand and so on.</p> <p>By convention, search APIs are exposed from a <code>GET</code> endpoint, and take parameters as query strings. Complex nested objects are usually serialized in bracket notation i.e. <code>?field[subfield]</code>:</p> <pre data-lang="http" data-highlighted="true" class="language-http z-code z-chroma"><code class="language-http" data-lang="http"><span class="z-err">GET /products? </span><span class="z-err"> filter[title][contains]=ssd&amp; </span><span class="z-err"> filter[price][lt]=100&amp; </span><span class="z-err"> sort[field]=price&amp; </span><span class="z-err"> sort[direction]=ASC </span></code></pre> <p>This is unnecessarily verbose. Field path prefixes are repeated multiple times. ASP.NET Core doesn't seem to support this bracketed format for binding complex types, either.</p> <p>I prefer querying the API with arguments serialized as JSON. It's more succinct, and also easier for clients to serialize.</p> <pre data-lang="http" data-highlighted="true" class="language-http z-code z-chroma"><code class="language-http" data-lang="http"><span class="z-err">GET /products?query= </span><span class="z-err"> { </span><span class="z-err"> &#34;filter&#34;: { </span><span class="z-err"> &#34;title&#34;: { </span><span class="z-err"> &#34;contains&#34;: &#34;ssd&#34; </span><span class="z-err"> }, </span><span class="z-err"> }, </span><span class="z-err"> &#34;sort&#34;: { </span><span class="z-err"> &#34;field&#34;: &#34;price&#34;, </span><span class="z-err"> &#34;direction&#34;: &#34;ASC&#34; </span><span class="z-err"> } </span><span class="z-err"> } </span></code></pre> <p>With ASP.NET Core, we get neither of them. To illustrate it, let's create an action that accepts a <code>SearchQuery</code> instance, populated from the query string:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[HttpGet]</span> <span class="z-kd">public</span> <span class="z-kd">async</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&lt;</span><span class="z-n">SearchResult</span><span class="z-p">&gt;&gt;</span> <span class="z-n">SearchProducts</span><span class="z-p">(</span> <span class="z-na"> [FromQuery]</span> <span class="z-n">SearchQuery</span> <span class="z-n">query</span> <span class="z-p">)</span> <span class="z-p">{</span> <span class="z-p">...</span> <span class="z-p">}</span></code></pre> <p>When we inspect the OpenAPI / Swagger schema, the action is described as having separate parameters for each property in my query type.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span> <span class="z-nt">&#34;paths&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;/api/products&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;get&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;parameters&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;name&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Filter.Title.Contains&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;in&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;query&#34;</span> <span class="z-err">/*</span> <span class="z-err">...</span> <span class="z-err">*/</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;name&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Sort.Field&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;in&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;query&#34;</span> <span class="z-err">/*</span> <span class="z-err">...</span> <span class="z-err">*/</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;name&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;Sort.Direction&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;in&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;query&#34;</span> <span class="z-err">/*</span> <span class="z-err">...</span> <span class="z-err">*/</span> <span class="z-p">}</span> <span class="z-p">]</span> <span class="z-c1">// ... </span><span class="z-p">}</span></code></pre> <p>This corresponds to a query string formatted in dot notation <code>field.subfield</code>:</p> <pre data-lang="http" data-highlighted="true" class="language-http z-code z-chroma"><code class="language-http" data-lang="http"><span class="z-err">GET /products? </span><span class="z-err"> Filter.Title.Contains=ssd&amp; </span><span class="z-err"> Sort.Field=price&amp; </span><span class="z-err"> Sort.Direction=ASC </span></code></pre> <p>and it is rendered as a form with separate inputs in Swagger UI.</p> <p><img src="/posts/aspnetcore-model-binding-json-query-params/./swagger_fromquery.png" alt="swagger ui with separate inputs" /></p> <p>If we try to send the parameter as JSON we can't get it to bind to a complex type. <code>SearchQuery query</code> parameter is always empty.</p> <p>We could <code>POST</code> this JSON in the request body, but that breaks the convention and doesn't signal that the endpoint is actually idempotent.</p> <h2 id="binding-json-query-parameters">Binding JSON query parameters</h2> <p>We can work around it by accepting a JSON string, then deserializing it inside the action.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-na">[HttpGet]</span> <span class="z-kd">public</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&lt;</span><span class="z-n">SearchResult</span><span class="z-p">&gt;&gt;</span> <span class="z-n">SearchProducts</span><span class="z-p">(</span> <span class="z-na"> [FromQuery(Name = &#34;query&#34;)]</span> <span class="z-kt">string</span> <span class="z-n">queryJson</span> <span class="z-p">)</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-n">query</span> <span class="z-p">=</span> <span class="z-n">DeserializeJson</span><span class="z-p">(</span><span class="z-n">queryJson</span><span class="z-p">);</span> <span class="z-c1">// ...</span> <span class="z-p">}</span></code></pre> <p>This works, but I don't like parsing, model binding &amp; validation myself that MVC platform already does for me. It'd be best if we could utilize ASP.NET as much as possible.</p> <p>MVC gives us a better extension point: model binders. We will use them to create our own binder and use it just like we would with <code>[FromQuery]</code>, or <code>[FromBody]</code>.</p> <h3 id="creating-a-custom-model-binder">Creating a custom model binder</h3> <p>We start with subclassing <code>ModelBinderAttribute</code>. It's a concrete class, so we don't have to implement anything, but we want to specify that we will be using a custom binder.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">internal</span> <span class="z-k">class</span> <span class="z-nc">FromJsonQueryAttribute</span> <span class="z-p">:</span> <span class="z-n">ModelBinderAttribute</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-n">FromJsonQueryAttribute</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-n">BinderType</span> <span class="z-p">=</span> <span class="z-k">typeof</span><span class="z-p">(</span><span class="z-n">JsonQueryBinder</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>We can annotate the action parameter with it directly instead of using <code>[ModelBinder]</code> attribute:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-n">Task</span><span class="z-p">&lt;</span><span class="z-n">ActionResult</span><span class="z-p">&lt;</span><span class="z-n">SearchResult</span><span class="z-p">&gt;&gt;</span> <span class="z-n">SearchProducts</span><span class="z-p">(</span> <span class="z-na"> [FromJsonQuery]</span> <span class="z-n">ProductySearchQuery</span> <span class="z-n">query</span> <span class="z-p">)</span> <span class="z-p">{</span> <span class="z-p">...</span> <span class="z-p">}</span></code></pre> <p>Now let's create the <code>JsonQueryBinder</code> class to get it working. It needs to implement <code>IModelBinder</code> interface. The steps we need to follow are:</p> <ul> <li> <p>Get raw JSON from the request.</p> <p><code>ModelBindingContext</code> contains <code>ValueProvider</code> for retrieving values from the request, if it's not enough, you can also use <code>HttpContext</code> and work on the request directly.</p> </li> <li> <p>Turn it into a useful type.</p> <p>We deserialize it with <a href="https://www.nuget.org/packages/System.Text.Json" target="_blank" rel="noopener">System.Text.Json</a>, which is available for all .NET versions, but you can't go wrong with <a href="https://www.nuget.org/packages/Newtonsoft.Json/" target="_blank" rel="noopener">Json.NET</a>, either.</p> </li> <li> <p>Validate it (<a href="#utilizing-asp-net-core-s-validation-tools">later</a>).</p> </li> <li> <p>Accept / reject the result.</p> </li> </ul> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">internal</span> <span class="z-k">class</span> <span class="z-nc">JsonQueryBinder</span> <span class="z-p">:</span> <span class="z-n">IModelBinder</span> <span class="z-p">{</span> <span class="z-kd">private</span> <span class="z-k">readonly</span> <span class="z-n">ILogger</span><span class="z-p">&lt;</span><span class="z-n">JsonQueryBinder</span><span class="z-p">&gt;</span> <span class="z-n">_logger</span><span class="z-p">;</span> <span class="z-kd">public</span> <span class="z-n">JsonQueryBinder</span><span class="z-p">(</span><span class="z-n">ILogger</span><span class="z-p">&lt;</span><span class="z-n">JsonQueryBinder</span><span class="z-p">&gt;</span> <span class="z-n">logger</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span> <span class="z-p">=</span> <span class="z-n">logger</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-n">Task</span> <span class="z-n">BindModelAsync</span><span class="z-p">(</span><span class="z-n">ModelBindingContext</span> <span class="z-n">bindingContext</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-k">value</span> <span class="z-p">=</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ValueProvider</span><span class="z-p">.</span><span class="z-n">GetValue</span><span class="z-p">(</span><span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">FieldName</span><span class="z-p">).</span><span class="z-n">FirstValue</span><span class="z-p">;</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-k">value</span> <span class="z-p">==</span> <span class="z-kc">null</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-n">Task</span><span class="z-p">.</span><span class="z-n">CompletedTask</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">try</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-n">parsed</span> <span class="z-p">=</span> <span class="z-n">JsonSerializer</span><span class="z-p">.</span><span class="z-n">Deserialize</span><span class="z-p">(</span> <span class="z-k">value</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ModelType</span><span class="z-p">,</span> <span class="z-k">new</span> <span class="z-n">JsonSerializerOptions</span><span class="z-p">(</span><span class="z-n">JsonSerializerDefaults</span><span class="z-p">.</span><span class="z-n">Web</span><span class="z-p">)</span> <span class="z-p">);</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">Result</span> <span class="z-p">=</span> <span class="z-n">ModelBindingResult</span><span class="z-p">.</span><span class="z-n">Success</span><span class="z-p">(</span><span class="z-n">parsed</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-n">Exception</span> <span class="z-n">e</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span><span class="z-p">.</span><span class="z-n">LogError</span><span class="z-p">(</span><span class="z-n">e</span><span class="z-p">,</span> <span class="z-s">&#34;Failed to bind &#39;{FieldName}&#39;&#34;</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">FieldName</span><span class="z-p">);</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">Result</span> <span class="z-p">=</span> <span class="z-n">ModelBindingResult</span><span class="z-p">.</span><span class="z-n">Failed</span><span class="z-p">();</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-n">Task</span><span class="z-p">.</span><span class="z-n">CompletedTask</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>Now when we send a request to the endpoint, it works! <code>SearchQuery query</code> parameter is filled correctly.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-err">GET</span> <span class="z-err">/products?query=</span><span class="z-p">{</span> <span class="z-nt">&#34;filter&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;title&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;contains&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;ssd&#34;</span> <span class="z-p">}</span> <span class="z-p">},</span> <span class="z-nt">&#34;sort&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;field&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;price&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;direction&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;ASC&#34;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>But validation doesn't work yet. If we omit a <code>[Required]</code> property or pass in a string to boolean, it just fails. We can improve this further.</p> <h3 id="utilizing-asp-net-core-s-validation-tools">Utilizing ASP.NET Core's validation tools</h3> <p>After we deserialize the payload, we can let ASP.NET take care of the validation. <a href="https://github.com/dotnet/aspnetcore/blob/85a6cb07ae2e1adf5a03740f8d40333b7b8e360c/src/Mvc/Mvc.Core/src/ModelBinding/Validation/IObjectModelValidator.cs#L11" target="_blank" rel="noopener"><code>IObjectModelValidator</code></a> interface gives us the tools to validate an object graph with all its properties and sub types.</p> <p>We inject it into our binder, and call its <code>.Validate()</code> method after deserialization.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">internal</span> <span class="z-k">class</span> <span class="z-nc">JsonQueryBinder</span> <span class="z-p">:</span> <span class="z-n">IModelBinder</span> <span class="z-p">{</span> <span class="z-c1">// inject the validator in constructor</span> <span class="z-kd">private</span> <span class="z-k">readonly</span> <span class="z-n">IObjectModelValidator</span> <span class="z-n">_validator</span><span class="z-p">;</span> <span class="z-c1">// ...</span> <span class="z-kd">public</span> <span class="z-n">Task</span> <span class="z-n">BindModelAsync</span><span class="z-p">(</span><span class="z-n">ModelBindingContext</span> <span class="z-n">bindingContext</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-k">try</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-n">parsed</span> <span class="z-p">=</span> <span class="z-n">JsonSerializer</span><span class="z-p">.</span><span class="z-n">Deserialize</span><span class="z-p">(</span><span class="z-k">value</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ModelType</span><span class="z-p">,</span> <span class="z-k">new</span> <span class="z-n">JsonSerializerOptions</span><span class="z-p">(</span><span class="z-n">JsonSerializerDefaults</span><span class="z-p">.</span><span class="z-n">Web</span><span class="z-p">));</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">Result</span> <span class="z-p">=</span> <span class="z-n">ModelBindingResult</span><span class="z-p">.</span><span class="z-n">Success</span><span class="z-p">(</span><span class="z-n">parsed</span><span class="z-p">);</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-n">parsed</span> <span class="z-p">!=</span> <span class="z-kc">null</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_validator</span><span class="z-p">.</span><span class="z-n">Validate</span><span class="z-p">(</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ActionContext</span><span class="z-p">,</span> <span class="z-n">validationState</span><span class="z-p">:</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ValidationState</span><span class="z-p">,</span> <span class="z-n">prefix</span><span class="z-p">:</span> <span class="z-kt">string</span><span class="z-p">.</span><span class="z-n">Empty</span><span class="z-p">,</span> <span class="z-n">model</span><span class="z-p">:</span> <span class="z-n">parsed</span> <span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-n">JsonException</span> <span class="z-n">e</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span><span class="z-p">.</span><span class="z-n">LogError</span><span class="z-p">(</span><span class="z-n">e</span><span class="z-p">,</span> <span class="z-s">&#34;Failed to bind parameter &#39;{FieldName}&#39;&#34;</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">FieldName</span><span class="z-p">);</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ActionContext</span><span class="z-p">.</span><span class="z-n">ModelState</span><span class="z-p">.</span><span class="z-n">TryAddModelError</span><span class="z-p">(</span><span class="z-n">key</span><span class="z-p">:</span> <span class="z-n">e</span><span class="z-p">.</span><span class="z-n">Path</span><span class="z-p">,</span> <span class="z-n">exception</span><span class="z-p">:</span> <span class="z-n">e</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ModelMetadata</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-n">Exception</span> <span class="z-n">e</span><span class="z-p">)</span> <span class="z-n">when</span> <span class="z-p">(</span><span class="z-n">e</span> <span class="z-k">is</span> <span class="z-n">FormatException</span> <span class="z-p">||</span> <span class="z-n">e</span> <span class="z-k">is</span> <span class="z-n">OverflowException</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span><span class="z-p">.</span><span class="z-n">LogError</span><span class="z-p">(</span><span class="z-n">e</span><span class="z-p">,</span> <span class="z-s">&#34;Failed to bind parameter &#39;{FieldName}&#39;&#34;</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">FieldName</span><span class="z-p">);</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ActionContext</span><span class="z-p">.</span><span class="z-n">ModelState</span><span class="z-p">.</span><span class="z-n">TryAddModelError</span><span class="z-p">(</span><span class="z-kt">string</span><span class="z-p">.</span><span class="z-n">Empty</span><span class="z-p">,</span> <span class="z-n">e</span><span class="z-p">,</span> <span class="z-n">bindingContext</span><span class="z-p">.</span><span class="z-n">ModelMetadata</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-n">Task</span><span class="z-p">.</span><span class="z-n">CompletedTask</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>Now if we have this annotated model:</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">record</span> <span class="z-nc">SearchSort</span><span class="z-p">(</span> <span class="z-na"> [Required]</span> <span class="z-kt">string</span> <span class="z-n">Field</span><span class="z-p">,</span> <span class="z-na"> [RegularExpression(&#34;ASC|DESC&#34;)]</span> <span class="z-kt">string</span> <span class="z-n">Direction</span> <span class="z-p">);</span></code></pre> <p>sending an invalid JSON returns a nicely formatted HTTP 400 error<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span> <span class="z-nt">&#34;type&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;https://tools.ietf.org/html/rfc7231#section-6.5.1&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;title&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;One or more validation errors occurred.&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;status&#34;</span><span class="z-p">:</span> <span class="z-mi">400</span><span class="z-p">,</span> <span class="z-nt">&#34;traceId&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;00-15833e541708ac49b4829f02c40008d0-4830117fc8dbca43-00&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;errors&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;Sort.Field&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-s2">&#34;The Field field is required.&#34;</span> <span class="z-p">],</span> <span class="z-nt">&#34;Sort.Direction&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-s2">&#34;The field Direction must match the regular expression &#39;ASC|DESC&#39;.&#34;</span> <span class="z-p">],</span> <span class="z-nt">&#34;q.Sort.Field&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-s2">&#34;The Field field is required.&#34;</span> <span class="z-p">],</span> <span class="z-nt">&#34;q.Sort.Direction&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-s2">&#34;The field Direction must match the regular expression &#39;ASC|DESC&#39;.&#34;</span> <span class="z-p">]</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <h2 id="adding-openapi-support">Adding OpenAPI support</h2> <p>When we created the <code>FromJsonQueryAttribute</code>, its <code>BindingSource</code> property was set to <a href="https://github.com/dotnet/aspnetcore/blob/85a6cb07ae2e1adf5a03740f8d40333b7b8e360c/src/Mvc/Mvc.Core/src/BindPropertyAttribute.cs#L65" target="_blank" rel="noopener"><code>BindingSource.Custom</code></a>. This keeps Swagger from displaying the properties as individual inputs.</p> <p><img src="/posts/aspnetcore-model-binding-json-query-params/./swagger_fromjsonquery.png" alt="swagger ui with single input" /></p> <p>But still, if we try to send a request using the Swagger UI, it serializes the parameter in bracket notation. To solve this problem, we can provide a custom API description for endpoints with <code>[FromJsonQuery]</code> parameters to tell clients to actually serialize the parameter as JSON.</p> <p>OpenAPI specification <a href="https://swagger.io/docs/specification/describing-parameters/#schema-vs-content" target="_blank" rel="noopener">addresses</a> our exact use-case, and gives us the option to specify a parameter schema with a specific content type like so:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span> <span class="z-nt">&#34;paths&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;/api/products&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;get&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;parameters&#34;</span><span class="z-p">:</span> <span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;name&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;query&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;in&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;query&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;content&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;application/json&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;schema&#34;</span><span class="z-p">:</span> <span class="z-p">{</span> <span class="z-nt">&#34;$ref&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;#/components/schemas/ProductSearchQuery&#34;</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-c1">// ... </span><span class="z-p">}</span></code></pre> <p>We can fix it by hooking into SwashBuckle with an <a href="https://github.com/domaindrivendev/Swashbuckle.AspNetCore#operation-filters" target="_blank" rel="noopener">operation filter</a> and modify the generated schema. From there, we correct the parameters annotated with <code>[FromJsonQuery]</code> with the right schema.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">internal</span> <span class="z-kd">static</span> <span class="z-k">class</span> <span class="z-nc">JsonQuerySwaggerGenExtensions</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-kd">static</span> <span class="z-n">SwaggerGenOptions</span> <span class="z-n">AddJsonQuerySupport</span><span class="z-p">(</span><span class="z-k">this</span> <span class="z-n">SwaggerGenOptions</span> <span class="z-n">options</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">options</span><span class="z-p">.</span><span class="z-n">OperationFilter</span><span class="z-p">&lt;</span><span class="z-n">JsonQueryOperationFilter</span><span class="z-p">&gt;();</span> <span class="z-k">return</span> <span class="z-n">options</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-kd">private</span> <span class="z-k">class</span> <span class="z-nc">JsonQueryOperationFilter</span> <span class="z-p">:</span> <span class="z-n">IOperationFilter</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">Apply</span><span class="z-p">(</span><span class="z-n">OpenApiOperation</span> <span class="z-n">operation</span><span class="z-p">,</span> <span class="z-n">OperationFilterContext</span> <span class="z-n">context</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-n">jsonQueryParams</span> <span class="z-p">=</span> <span class="z-n">context</span><span class="z-p">.</span><span class="z-n">ApiDescription</span><span class="z-p">.</span><span class="z-n">ActionDescriptor</span><span class="z-p">.</span><span class="z-n">Parameters</span> <span class="z-p">.</span><span class="z-n">Where</span><span class="z-p">(</span><span class="z-n">ad</span> <span class="z-p">=&gt;</span> <span class="z-n">ad</span><span class="z-p">.</span><span class="z-n">BindingInfo</span><span class="z-p">.</span><span class="z-n">BinderType</span> <span class="z-p">==</span> <span class="z-k">typeof</span><span class="z-p">(</span><span class="z-n">JsonQueryBinder</span><span class="z-p">))</span> <span class="z-p">.</span><span class="z-n">Select</span><span class="z-p">(</span><span class="z-n">ad</span> <span class="z-p">=&gt;</span> <span class="z-n">ad</span><span class="z-p">.</span><span class="z-n">Name</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">ToList</span><span class="z-p">();</span> <span class="z-k">if</span> <span class="z-p">(!</span><span class="z-n">jsonQueryParams</span><span class="z-p">.</span><span class="z-n">Any</span><span class="z-p">())</span> <span class="z-p">{</span> <span class="z-k">return</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">foreach</span> <span class="z-p">(</span><span class="z-kt">var</span> <span class="z-n">p</span> <span class="z-k">in</span> <span class="z-n">operation</span><span class="z-p">.</span><span class="z-n">Parameters</span><span class="z-p">.</span><span class="z-n">Where</span><span class="z-p">(</span><span class="z-n">p</span> <span class="z-p">=&gt;</span> <span class="z-n">jsonQueryParams</span><span class="z-p">.</span><span class="z-n">Contains</span><span class="z-p">(</span><span class="z-n">p</span><span class="z-p">.</span><span class="z-n">Name</span><span class="z-p">)))</span> <span class="z-p">{</span> <span class="z-c1">// move the schema under application/json content type</span> <span class="z-n">p</span><span class="z-p">.</span><span class="z-n">Content</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">Dictionary</span><span class="z-p">&lt;</span><span class="z-kt">string</span><span class="z-p">,</span> <span class="z-n">OpenApiMediaType</span><span class="z-p">&gt;()</span> <span class="z-p">{</span> <span class="z-na"> [MediaTypeNames.Application.Json]</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">OpenApiMediaType</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-n">Schema</span> <span class="z-p">=</span> <span class="z-n">p</span><span class="z-p">.</span><span class="z-n">Schema</span> <span class="z-p">}</span> <span class="z-p">};</span> <span class="z-c1">// then clear it</span> <span class="z-n">p</span><span class="z-p">.</span><span class="z-n">Schema</span> <span class="z-p">=</span> <span class="z-kc">null</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>When writing a plugin for a third party library, I prefer having an extension method to hide the glue code, and keep the implementations as nested private classes.</p> <p>Anyway, now we can enable the filter in our <code>Startup</code> class.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">ConfigureServices</span><span class="z-p">(</span><span class="z-n">IServiceCollection</span> <span class="z-n">services</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddSwaggerGen</span><span class="z-p">(</span><span class="z-n">c</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-n">c</span><span class="z-p">.</span><span class="z-n">AddJsonQuerySupport</span><span class="z-p">();</span> <span class="z-c1">// ...</span> <span class="z-p">});</span> <span class="z-p">}</span></code></pre> <p>When we send a request, the query parameter is now serialized as JSON 💃. Our API is now easier to use, well-annotated and shows up with the correct Swagger UI for testing.</p> <p>Cheers ✌.</p> <h2 id="references">References</h2> <ul> <li><a href="https://docs.microsoft.com/en-us/aspnet/core/mvc/models/model-binding?view=aspnetcore-5.0" target="_blank" rel="noopener">Model Binding in ASP.NET Core</a></li> <li><a href="https://docs.microsoft.com/en-us/aspnet/core/mvc/advanced/custom-model-binding?view=aspnetcore-5.0#custom-model-binder-sample" target="_blank" rel="noopener">Custom model binder</a></li> <li><a href="https://github.com/domaindrivendev/Swashbuckle.AspNetCore" target="_blank" rel="noopener">SwashBuckle</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>You need to annotate your controller with <code>[ApiController]</code> attribute to get auto-validation, along with other useful behaviors for APIs. See <a href="https://docs.microsoft.com/en-us/aspnet/core/web-api/?view=aspnetcore-5.0#apicontroller-attribute" target="_blank" rel="noopener">docs</a> for further info.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Tue, 26 Jan 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/checking-executable-exists-in-path-using-node/ https://abdus.dev/posts/checking-executable-exists-in-path-using-node/ <p>I was looking for a way to distribute a Powershell script. I decided to wrap it inside an npm package and let users <a href="/posts/running-powershell-script-in-node/">run the Powershell script</a> using:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">npx</span> <span class="z-nb">package-name</span> <span class="z-p">[...</span><span class="z-n">args</span><span class="z-p">]</span></code></pre> <p>which passes arguments to Powershell and let it run the actual script, which is bundled with the npm package.</p> <h2 id="problem">Problem</h2> <p>The problem arised when I wanted to check if a modern version of Powershell exists on the user's machine. The first solution that came to my mind was checking executable version<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">pwsh</span> <span class="z-n">-command</span> <span class="z-s2">&#34;echo </span><span class="z-nv">$PSVersionTable</span><span class="z-s2">.PSVersion.ToString()&#34;</span> </code></pre> <p>but it runs slowly. It also doesn't handle commands without <code>--version</code> or any command switch at all. So trying to run the program to see if actually exists is not a great way to go about it.</p> <h2 id="solution">Solution</h2> <p>A faster solution is to check if <code>pwsh.exe</code> exists in <code>PATH</code>. There's a pre-made solution for this, as with anything, called <a href="https://github.com/springernature/hasbin" target="_blank" rel="noopener">hasbin</a> but it's quite old and uses external dependencies for async operations that are native to modern JS.</p> <p><code>findExecutable</code> function builds a list of candidates paths from the combinations of all directories in <code>PATH</code> and all extensions (no-op if not on Windows) and checks if any of them exists and returns the first result.</p> <p>If I were to look for <code>pwsh.exe</code> and <code>PATH</code> were <code>a/b;c/d</code>, it would try <code>a/b/pwsh.exe</code>, <code>c/d/pwsh.exe</code> and so on.</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-kr">const</span> <span class="z-nx">path</span> <span class="z-o">=</span> <span class="z-nx">require</span><span class="z-p">(</span><span class="z-s2">&#34;path&#34;</span><span class="z-p">);</span> <span class="z-kr">const</span> <span class="z-nx">fs</span> <span class="z-o">=</span> <span class="z-nx">require</span><span class="z-p">(</span><span class="z-s2">&#34;fs/promises&#34;</span><span class="z-p">);</span> <span class="z-cm">/** </span><span class="z-cm"> * @param {string} exe executable name (without extension if on Windows) </span><span class="z-cm"> * @return {Promise&lt;string|null&gt;} executable path if found </span><span class="z-cm"> * */</span> <span class="z-kr">async</span> <span class="z-kd">function</span> <span class="z-nx">findExecutable</span><span class="z-p">(</span><span class="z-nx">exe</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-kr">const</span> <span class="z-nx">envPath</span> <span class="z-o">=</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">env</span><span class="z-p">.</span><span class="z-nx">PATH</span> <span class="z-o">||</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">;</span> <span class="z-kr">const</span> <span class="z-nx">envExt</span> <span class="z-o">=</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">env</span><span class="z-p">.</span><span class="z-nx">PATHEXT</span> <span class="z-o">||</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">;</span> <span class="z-kr">const</span> <span class="z-nx">pathDirs</span> <span class="z-o">=</span> <span class="z-nx">envPath</span> <span class="z-p">.</span><span class="z-nx">replace</span><span class="z-p">(</span><span class="z-sr">/[&#34;]+/g</span><span class="z-p">,</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-nx">split</span><span class="z-p">(</span><span class="z-nx">path</span><span class="z-p">.</span><span class="z-nx">delimiter</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-nx">filter</span><span class="z-p">(</span><span class="z-nb">Boolean</span><span class="z-p">);</span> <span class="z-kr">const</span> <span class="z-nx">extensions</span> <span class="z-o">=</span> <span class="z-nx">envExt</span><span class="z-p">.</span><span class="z-nx">split</span><span class="z-p">(</span><span class="z-s2">&#34;;&#34;</span><span class="z-p">);</span> <span class="z-kr">const</span> <span class="z-nx">candidates</span> <span class="z-o">=</span> <span class="z-nx">pathDirs</span><span class="z-p">.</span><span class="z-nx">flatMap</span><span class="z-p">((</span><span class="z-nx">d</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-nx">extensions</span><span class="z-p">.</span><span class="z-nx">map</span><span class="z-p">((</span><span class="z-nx">ext</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-nx">path</span><span class="z-p">.</span><span class="z-nx">join</span><span class="z-p">(</span><span class="z-nx">d</span><span class="z-p">,</span> <span class="z-nx">exe</span> <span class="z-o">+</span> <span class="z-nx">ext</span><span class="z-p">))</span> <span class="z-p">);</span> <span class="z-k">try</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-kr">await</span> <span class="z-nb">Promise</span><span class="z-p">.</span><span class="z-nx">any</span><span class="z-p">(</span><span class="z-nx">candidates</span><span class="z-p">.</span><span class="z-nx">map</span><span class="z-p">(</span><span class="z-nx">checkFileExists</span><span class="z-p">));</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-kc">null</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-kr">async</span> <span class="z-kd">function</span> <span class="z-nx">checkFileExists</span><span class="z-p">(</span><span class="z-nx">filePath</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">if</span> <span class="z-p">((</span><span class="z-kr">await</span> <span class="z-nx">fs</span><span class="z-p">.</span><span class="z-nx">stat</span><span class="z-p">(</span><span class="z-nx">filePath</span><span class="z-p">)).</span><span class="z-nx">isFile</span><span class="z-p">())</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-nx">filePath</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">throw</span> <span class="z-k">new</span> <span class="z-nb">Error</span><span class="z-p">(</span><span class="z-s2">&#34;Not a file&#34;</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>We can finally use <code>findExecutable</code> to get the most recent Powershell installation.</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-kr">async</span> <span class="z-kd">function</span> <span class="z-nx">getPowershellVariant</span><span class="z-p">()</span> <span class="z-p">{</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-kr">await</span> <span class="z-nx">findExecutable</span><span class="z-p">(</span><span class="z-s2">&#34;pwsh&#34;</span><span class="z-p">))</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-s2">&#34;pwsh&#34;</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-kr">await</span> <span class="z-nx">findExecutable</span><span class="z-p">(</span><span class="z-s2">&#34;powershell&#34;</span><span class="z-p">))</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-s2">&#34;powershell&#34;</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-k">throw</span> <span class="z-k">new</span> <span class="z-nb">Error</span><span class="z-p">(</span><span class="z-s2">&#34;Powershell is not installed, or not in PATH&#34;</span><span class="z-p">);</span> <span class="z-p">}</span></code></pre> <h2 id="what-is-pathext-for">What is <code>PATHEXT</code> for?</h2> <p>Windows uses <code>PATHEXT</code> environment variable to determine the file name for an executable if you don't specify an extension. On my PC, <code>PATHEXT</code> content is:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-nv">$env:PATHEXT</span> <span class="z-p">.</span><span class="z-n">COM</span><span class="z-p">;.</span><span class="z-n">EXE</span><span class="z-p">;.</span><span class="z-n">BAT</span><span class="z-p">;.</span><span class="z-n">CMD</span><span class="z-p">;.</span><span class="z-n">VBS</span><span class="z-p">;.</span><span class="z-n">VBE</span><span class="z-p">;.</span><span class="z-n">JS</span><span class="z-p">;.</span><span class="z-n">JSE</span><span class="z-p">;.</span><span class="z-n">WSF</span><span class="z-p">;.</span><span class="z-n">WSH</span><span class="z-p">;.</span><span class="z-n">MSC</span><span class="z-p">;.</span><span class="z-n">PY</span><span class="z-p">;.</span><span class="z-n">PYW</span><span class="z-p">;.</span><span class="z-n">ps1</span><span class="z-p">;.</span><span class="z-py">CPL</span></code></pre> <p>That means, if I just type <code>pwsh</code>, it looks for <code>pwsh.com</code>, <code>pwsh.exe</code>, <code>pwsh.bat</code> and so on, and executes the first one it finds. That also means if an extension is not on the list, I have to type in the extension.</p> <p>This is not an issue on UNIX systems, because they use an <a href="https://en.wikipedia.org/wiki/File-system_permissions#Symbolic_notation" target="_blank" rel="noopener">executable bit <code>+x</code></a> to specify if a file can be executed, regardless of whether it has an extension or not.</p> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>This usually involves running a binary with <code>--version</code> option, but Powershell behaves completely differently and uses <code>-version</code> option to specify the version of Powershell to run the command/script against.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Thu, 21 Jan 2021 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/running-powershell-script-in-node/ https://abdus.dev/posts/running-powershell-script-in-node/ <p>I needed a way to call a Powershell script from Node. Node provides <code>child_process</code> module, which provides tools to execute and communicate with external processes.</p> <p>We're going to use <code>spawn</code> for this. It allows us to pipe the <code>stdout</code>/<code>stderr</code> of a process back to caller. This helps the caller to see what the process is actually doing behind the scenes.</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-kr">const</span> <span class="z-p">{</span><span class="z-nx">spawn</span><span class="z-p">}</span> <span class="z-o">=</span> <span class="z-nx">require</span><span class="z-p">(</span><span class="z-s2">&#34;child_process&#34;</span><span class="z-p">);</span> <span class="z-cm">/** </span><span class="z-cm"> * @param {string} executable </span><span class="z-cm"> * @param {string[]} args </span><span class="z-cm"> * @param {import(&#39;child_process&#39;).SpawnOptions} opts </span><span class="z-cm"> * @return {Promise&lt;number&gt;} return code </span><span class="z-cm"> * */</span> <span class="z-kr">async</span> <span class="z-kd">function</span> <span class="z-nx">run</span><span class="z-p">(</span><span class="z-nx">executable</span><span class="z-p">,</span> <span class="z-nx">args</span><span class="z-p">,</span> <span class="z-nx">opts</span> <span class="z-o">=</span> <span class="z-p">{})</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-k">new</span> <span class="z-nb">Promise</span><span class="z-p">((</span><span class="z-nx">resolve</span><span class="z-p">,</span> <span class="z-nx">reject</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-kr">const</span> <span class="z-nx">child</span> <span class="z-o">=</span> <span class="z-nx">spawn</span><span class="z-p">(</span><span class="z-nx">executable</span><span class="z-p">,</span> <span class="z-nx">args</span><span class="z-p">,</span> <span class="z-p">{</span> <span class="z-nx">shell</span><span class="z-o">:</span> <span class="z-kc">true</span><span class="z-p">,</span> <span class="z-nx">stdio</span><span class="z-o">:</span> <span class="z-p">[</span><span class="z-s2">&#34;pipe&#34;</span><span class="z-p">,</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">stdout</span><span class="z-p">,</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">stderr</span><span class="z-p">],</span> <span class="z-p">...</span><span class="z-nx">opts</span><span class="z-p">,</span> <span class="z-p">});</span> <span class="z-nx">child</span><span class="z-p">.</span><span class="z-nx">on</span><span class="z-p">(</span><span class="z-s2">&#34;error&#34;</span><span class="z-p">,</span> <span class="z-nx">reject</span><span class="z-p">);</span> <span class="z-nx">child</span><span class="z-p">.</span><span class="z-nx">on</span><span class="z-p">(</span><span class="z-s2">&#34;exit&#34;</span><span class="z-p">,</span> <span class="z-p">(</span><span class="z-nx">code</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-nx">code</span> <span class="z-o">===</span> <span class="z-mi">0</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nx">resolve</span><span class="z-p">(</span><span class="z-nx">code</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-k">else</span> <span class="z-p">{</span> <span class="z-kr">const</span> <span class="z-nx">e</span> <span class="z-o">=</span> <span class="z-k">new</span> <span class="z-nb">Error</span><span class="z-p">(</span><span class="z-s1">&#39;Process exited with error code &#39;</span> <span class="z-o">+</span> <span class="z-nx">code</span><span class="z-p">);</span> <span class="z-nx">e</span><span class="z-p">.</span><span class="z-nx">code</span> <span class="z-o">=</span> <span class="z-nx">code</span><span class="z-p">;</span> <span class="z-nx">reject</span><span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-p">});</span> <span class="z-p">});</span> <span class="z-p">}</span></code></pre> <p>I like writing my own wrapper to promisify <code>spawn</code>. It helps me return the exit code to the caller and I also get a better intellisense while writing scripts.</p> <p>We can call <code>run</code> inside a try-catch block to capture errors and set a correct exit code.</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-k">try</span> <span class="z-p">{</span> <span class="z-kr">const</span> <span class="z-nx">code</span> <span class="z-o">=</span> <span class="z-kr">await</span> <span class="z-nx">run</span><span class="z-p">(</span><span class="z-s1">&#39;powershell&#39;</span><span class="z-p">,</span> <span class="z-p">[</span><span class="z-s2">&#34;-executionpolicy&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;unrestricted&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;-file&#34;</span><span class="z-p">,</span> <span class="z-s1">&#39;script.ps1&#39;</span><span class="z-p">]);</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">exit</span><span class="z-p">(</span><span class="z-nx">code</span><span class="z-p">);</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nx">console</span><span class="z-p">.</span><span class="z-nx">error</span><span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">);</span> <span class="z-nx">process</span><span class="z-p">.</span><span class="z-nx">exit</span><span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">.</span><span class="z-nx">code</span> <span class="z-o">||</span> <span class="z-mi">1</span><span class="z-p">);</span> <span class="z-p">}</span></code></pre> Mon, 30 Nov 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/monitor-clipboard/ https://abdus.dev/posts/monitor-clipboard/ <p>I was looking for ways to listen to the clipboard and get called for updates as its content changes. I found a couple of ways to achieve this in Python. Some solutions poll for changes, others use ctypes and Win32 APIs.</p> <p>Working with C bindings in Python is frustrating. The debugger doesn't work well with pointers and C types. You have to build your own C structs to unpack a pointer. Win32 APIs are no exception, as they're written in C/C++ and Python is probably not the best language to use them. But still, it works well enough for our purposes.</p> <p>We'll explore how do to it in Python using Win32 APIs, or alternatively integrating <a href="#using-dumpclip-polling-for-changes">a utility</a> I've written in C# that retrieves clipboard contents or streams updates to it. I had written this earlier when I didn't know much about Win32 APIs or how to use them, but it still functions well, so I'm leaving it here in this post as reference.</p> <p>To make working with Win32 APIs easier, we need to install <a href="https://github.com/mhammond/pywin32" target="_blank" rel="noopener"><code>pywin32</code></a> package which provides most of the primitives and types for Win32 APIs, though it's not a strict dependency.</p> <h2 id="monitoring-clipboard-updates">Monitoring clipboard updates</h2> <p>Windows provides a couple of methods for data exchange between applications. Clipboard is one of them. All applications have access to it. But we first need to create a primitive &quot;application&quot; that Windows recognizes. We subscribe it for the clipboard updates.</p> <div class="has-recommend"> <p>Windows uses <strong>window</strong>s (hah!) as the building block of applications. I've written about how windows and messaging works on Windows in <a href="/posts/python-monitor-usb/#listening-to-windows-wm-devicechange-messages">another post</a> where I explored USB hotplugging events, which might be worth reading.</p> <div class="recommend"> <p><a href="/posts/python-monitor-usb/#listening-to-windows-wm-devicechange-messages">Detecting USB drive insertion &amp; removal on Windows using Python</a></p> </div> </div> <p>Let's create a window, and set <code>print</code> function as its <a href="https://docs.microsoft.com/en-us/windows/win32/winmsg/using-window-procedures" target="_blank" rel="noopener">window procedure</a>:</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">win32api</span><span class="z-o">,</span> <span class="z-nn">win32gui</span> <span class="z-k">def</span> <span class="z-nf">create_window</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-nb">int</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Create a window for listening to messages </span><span class="z-s2"> :return: window hwnd </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">wc</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">WNDCLASS</span><span class="z-p">()</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpfnWndProc</span> <span class="z-o">=</span> <span class="z-nb">print</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpszClassName</span> <span class="z-o">=</span> <span class="z-s1">&#39;demo&#39;</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span> <span class="z-o">=</span> <span class="z-n">win32api</span><span class="z-o">.</span><span class="z-n">GetModuleHandle</span><span class="z-p">(</span><span class="z-kc">None</span><span class="z-p">)</span> <span class="z-n">class_atom</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">RegisterClass</span><span class="z-p">(</span><span class="z-n">wc</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">CreateWindow</span><span class="z-p">(</span><span class="z-n">class_atom</span><span class="z-p">,</span> <span class="z-s1">&#39;demo&#39;</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span><span class="z-p">,</span> <span class="z-kc">None</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">hwnd</span> <span class="z-o">=</span> <span class="z-n">create_window</span><span class="z-p">()</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span></code></pre> <p>When we run it doesn't do much except to dump messages sent by Windows to console. We receive the first message <code>WM_DWMNCRENDERINGCHANGED</code>, which doesn't concern us.</p> <p>We need to register this window as a &quot;clipboard format listener&quot; using <a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/using-the-clipboard#creating-a-clipboard-format-listener" target="_blank" rel="noopener"><code>AddClipboardFormatListener</code></a> API, to get notified by Windows whenever the contents of the clipboard change.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">ctypes</span> <span class="z-c1"># ...</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">hwnd</span> <span class="z-o">=</span> <span class="z-n">create_window</span><span class="z-p">()</span> <span class="z-n">ctypes</span><span class="z-o">.</span><span class="z-n">windll</span><span class="z-o">.</span><span class="z-n">user32</span><span class="z-o">.</span><span class="z-n">AddClipboardFormatListener</span><span class="z-p">(</span><span class="z-n">hwnd</span><span class="z-p">)</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span></code></pre> <p>Now when we run this, it still prints the previous message, but when you copy something to the clipboard it receives another message:</p> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">2033456 799 1 0 </span><span class="z-go">2033456 797 8 0 </span></code></pre> <p>Decoding the second message:</p> <table> <thead> <tr> <th>Value</th> <th>Hex</th> <th>Message</th> </tr> </thead> <tbody> <tr> <td><code>797</code></td> <td><code>0x031D</code></td> <td><code>WM_CLIPBOARDUPDATE</code> 🥳</td> </tr> </tbody> </table> <p>We've received a <a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/wm-clipboardupdate" target="_blank" rel="noopener"><code>WM_CLIPBOARDUPDATE</code></a> message notifying us that the clipboard content has changed. Now we can build our script around it.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">threading</span> <span class="z-kn">import</span> <span class="z-nn">ctypes</span> <span class="z-kn">import</span> <span class="z-nn">win32api</span><span class="z-o">,</span> <span class="z-nn">win32gui</span> <span class="z-k">class</span> <span class="z-nc">Clipboard</span><span class="z-p">:</span> <span class="z-k">def</span> <span class="z-nf">_create_window</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-nb">int</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Create a window for listening to messages </span><span class="z-s2"> :return: window hwnd </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">wc</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">WNDCLASS</span><span class="z-p">()</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpfnWndProc</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_process_message</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpszClassName</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span> <span class="z-o">=</span> <span class="z-n">win32api</span><span class="z-o">.</span><span class="z-n">GetModuleHandle</span><span class="z-p">(</span><span class="z-kc">None</span><span class="z-p">)</span> <span class="z-n">class_atom</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">RegisterClass</span><span class="z-p">(</span><span class="z-n">wc</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">CreateWindow</span><span class="z-p">(</span><span class="z-n">class_atom</span><span class="z-p">,</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span><span class="z-p">,</span> <span class="z-kc">None</span><span class="z-p">)</span> <span class="z-k">def</span> <span class="z-nf">_process_message</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">,</span> <span class="z-n">hwnd</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">msg</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">wparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">lparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">):</span> <span class="z-n">WM_CLIPBOARDUPDATE</span> <span class="z-o">=</span> <span class="z-mh">0x031D</span> <span class="z-k">if</span> <span class="z-n">msg</span> <span class="z-o">==</span> <span class="z-n">WM_CLIPBOARDUPDATE</span><span class="z-p">:</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-s1">&#39;clipboard updated!&#39;</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-mi">0</span> <span class="z-k">def</span> <span class="z-nf">listen</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">):</span> <span class="z-k">def</span> <span class="z-nf">runner</span><span class="z-p">():</span> <span class="z-n">hwnd</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_create_window</span><span class="z-p">()</span> <span class="z-n">ctypes</span><span class="z-o">.</span><span class="z-n">windll</span><span class="z-o">.</span><span class="z-n">user32</span><span class="z-o">.</span><span class="z-n">AddClipboardFormatListener</span><span class="z-p">(</span><span class="z-n">hwnd</span><span class="z-p">)</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span> <span class="z-n">th</span> <span class="z-o">=</span> <span class="z-n">threading</span><span class="z-o">.</span><span class="z-n">Thread</span><span class="z-p">(</span><span class="z-n">target</span><span class="z-o">=</span><span class="z-n">runner</span><span class="z-p">,</span> <span class="z-n">daemon</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span> <span class="z-k">while</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">is_alive</span><span class="z-p">():</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">(</span><span class="z-mf">0.25</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">clipboard</span> <span class="z-o">=</span> <span class="z-n">Clipboard</span><span class="z-p">()</span> <span class="z-n">clipboard</span><span class="z-o">.</span><span class="z-n">listen</span><span class="z-p">()</span></code></pre> <p>One thing we need to watch out for is that because <code>win32gui.PumpMessages()</code> is blocking, we cannot stop the script using <kbd>Ctrl</kbd> + <kbd>C</kbd>. So we run it inside a thread, which lets <code>KeyboardInterrupt</code> to bubble up and terminate the script.</p> <p>When we run it, and copy something (text, files) and check the console, we can see it prints <code>clipboard updated!</code>.</p> <p>Now that we have the notification working, let's retrieve the what's actually in the clipboard.</p> <h2 id="getting-clipboard-contents">Getting clipboard contents</h2> <p><a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/using-the-clipboard#creating-a-clipboard-format-listener" target="_blank" rel="noopener">Windows clipboard</a> has a concept called <a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/standard-clipboard-formats#constants" target="_blank" rel="noopener">&quot;clipboard format&quot;</a>. When you copy something, (depending on application) the payload is also attached a bunch of metadata, allowing it to be used in various contexts. For example, when you copy a table from a webpage, you have the option to paste it as plain text, or paste it in Excel and have it formatted as a table. You can copy files, images, screenshots into the clipboard and each payload gets stored formatted (again, depending on how the application sets the clipboard content).</p> <p>Therefore, if we want to get the clipboard contents, we need to specify which format we want in. For now, we'll be dealing with:</p> <div class="table"> <table> <thead> <tr> <th>Format</th> <th>Value</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>CF_UNICODETEXT</code></td> <td><code>13</code></td> <td>Unicode text format</td> </tr> <tr> <td><code>CF_TEXT</code></td> <td><code>1</code></td> <td>Text format for ANSI text</td> </tr> <tr> <td><code>CF_HDROP</code></td> <td><code>15</code></td> <td>List of files</td> </tr> <tr> <td><code>CF_BITMAP</code></td> <td><code>2</code></td> <td>Images e.g. screenshots</td> </tr> </tbody> </table> </div> <p>To read the clipboard, we'll use <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Winuser/nf-winuser-openclipboard" target="_blank" rel="noopener"><code>OpenClipboard</code></a> to set a lock first. This ensures other programs can't modify the clipboard while we're trying to read it. We need to release the lock with <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Winuser/nf-winuser-closeclipboard" target="_blank" rel="noopener"><code>CloseClipboard</code></a> once we're done.</p> <p>Then we'll call <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Winuser/nf-winuser-isclipboardformatavailable" target="_blank" rel="noopener"><code>IsClipboardFormatAvailable</code></a> to query a format, then get its contents using <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Winuser/nf-winuser-getclipboarddata" target="_blank" rel="noopener"><code>GetClipboardData</code></a>, or fallback to other formats.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">from</span> <span class="z-nn">pathlib</span> <span class="z-kn">import</span> <span class="z-n">Path</span> <span class="z-kn">from</span> <span class="z-nn">dataclasses</span> <span class="z-kn">import</span> <span class="z-n">dataclass</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Union</span><span class="z-p">,</span> <span class="z-n">List</span><span class="z-p">,</span> <span class="z-n">Optional</span> <span class="z-kn">import</span> <span class="z-nn">win32clipboard</span><span class="z-o">,</span> <span class="z-nn">win32con</span> <span class="z-nd">@dataclass</span> <span class="z-k">class</span> <span class="z-nc">Clip</span><span class="z-p">:</span> <span class="z-nb">type</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">value</span><span class="z-p">:</span> <span class="z-n">Union</span><span class="z-p">[</span><span class="z-nb">str</span><span class="z-p">,</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Path</span><span class="z-p">]]</span> <span class="z-k">def</span> <span class="z-nf">read_clipboard</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">Optional</span><span class="z-p">[</span><span class="z-n">Clip</span><span class="z-p">]:</span> <span class="z-k">try</span><span class="z-p">:</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">OpenClipboard</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">IsClipboardFormatAvailable</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_HDROP</span><span class="z-p">):</span> <span class="z-n">data</span><span class="z-p">:</span> <span class="z-nb">tuple</span> <span class="z-o">=</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">GetClipboardData</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_HDROP</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;files&#39;</span><span class="z-p">,</span> <span class="z-p">[</span><span class="z-n">Path</span><span class="z-p">(</span><span class="z-n">f</span><span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">f</span> <span class="z-ow">in</span> <span class="z-n">data</span><span class="z-p">])</span> <span class="z-k">elif</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">IsClipboardFormatAvailable</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_UNICODETEXT</span><span class="z-p">):</span> <span class="z-n">data</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-o">=</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">GetClipboardData</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_UNICODETEXT</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;text&#39;</span><span class="z-p">,</span> <span class="z-n">data</span><span class="z-p">)</span> <span class="z-k">elif</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">IsClipboardFormatAvailable</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_TEXT</span><span class="z-p">):</span> <span class="z-n">data</span><span class="z-p">:</span> <span class="z-nb">bytes</span> <span class="z-o">=</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">GetClipboardData</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_TEXT</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;text&#39;</span><span class="z-p">,</span> <span class="z-n">data</span><span class="z-o">.</span><span class="z-n">decode</span><span class="z-p">())</span> <span class="z-k">elif</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">IsClipboardFormatAvailable</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_BITMAP</span><span class="z-p">):</span> <span class="z-c1"># TODO: handle screenshots</span> <span class="z-k">pass</span> <span class="z-k">return</span> <span class="z-kc">None</span> <span class="z-k">finally</span><span class="z-p">:</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">CloseClipboard</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-n">read_clipboard</span><span class="z-p">())</span></code></pre> <p>When we run it, and try copying some text or files, it prints the contents to the console:</p> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">Clip(type=&#39;text&#39;, value=&#39;read_clipboard&#39;) </span><span class="z-go">Clip(type=&#39;files&#39;, value=[WindowsPath(&#39;C:/Python39/vcruntime140_1.dll&#39;), WindowsPath(&#39;C:/Python39/python.exe&#39;)]) </span></code></pre> <p>Now let's bring it all together:</p> <h2 id="clipboard-listener-in-python">Clipboard listener in Python</h2> <p>I've placed <code>read_clipboard</code> inside <code>Clipboard</code> class, which creates a window and subscribes to clipboard updates. When the clipboard content changes, it triggers suitable callbacks with the parsed content.</p> <p>For convenience, you can enable <code>trigger_at_start</code> to trigger callbacks with the current clipboard content immediately after listening.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">ctypes</span> <span class="z-kn">import</span> <span class="z-nn">threading</span> <span class="z-kn">from</span> <span class="z-nn">dataclasses</span> <span class="z-kn">import</span> <span class="z-n">dataclass</span> <span class="z-kn">from</span> <span class="z-nn">pathlib</span> <span class="z-kn">import</span> <span class="z-n">Path</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span><span class="z-p">,</span> <span class="z-n">Union</span><span class="z-p">,</span> <span class="z-n">List</span><span class="z-p">,</span> <span class="z-n">Optional</span> <span class="z-kn">import</span> <span class="z-nn">win32api</span><span class="z-o">,</span> <span class="z-nn">win32clipboard</span><span class="z-o">,</span> <span class="z-nn">win32con</span><span class="z-o">,</span> <span class="z-nn">win32gui</span> <span class="z-k">class</span> <span class="z-nc">Clipboard</span><span class="z-p">:</span> <span class="z-nd">@dataclass</span> <span class="z-k">class</span> <span class="z-nc">Clip</span><span class="z-p">:</span> <span class="z-nb">type</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">value</span><span class="z-p">:</span> <span class="z-n">Union</span><span class="z-p">[</span><span class="z-nb">str</span><span class="z-p">,</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Path</span><span class="z-p">]]</span> <span class="z-k">def</span> <span class="z-fm">__init__</span><span class="z-p">(</span> <span class="z-bp">self</span><span class="z-p">,</span> <span class="z-n">trigger_at_start</span><span class="z-p">:</span> <span class="z-nb">bool</span> <span class="z-o">=</span> <span class="z-kc">False</span><span class="z-p">,</span> <span class="z-n">on_text</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-nb">str</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">]</span> <span class="z-o">=</span> <span class="z-kc">None</span><span class="z-p">,</span> <span class="z-n">on_update</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-n">Clip</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">]</span> <span class="z-o">=</span> <span class="z-kc">None</span><span class="z-p">,</span> <span class="z-n">on_files</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-nb">str</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">]</span> <span class="z-o">=</span> <span class="z-kc">None</span><span class="z-p">,</span> <span class="z-p">):</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_trigger_at_start</span> <span class="z-o">=</span> <span class="z-n">trigger_at_start</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_update</span> <span class="z-o">=</span> <span class="z-n">on_update</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_files</span> <span class="z-o">=</span> <span class="z-n">on_files</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_text</span> <span class="z-o">=</span> <span class="z-n">on_text</span> <span class="z-k">def</span> <span class="z-nf">_create_window</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-nb">int</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Create a window for listening to messages </span><span class="z-s2"> :return: window hwnd </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">wc</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">WNDCLASS</span><span class="z-p">()</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpfnWndProc</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_process_message</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpszClassName</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span> <span class="z-o">=</span> <span class="z-n">win32api</span><span class="z-o">.</span><span class="z-n">GetModuleHandle</span><span class="z-p">(</span><span class="z-kc">None</span><span class="z-p">)</span> <span class="z-n">class_atom</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">RegisterClass</span><span class="z-p">(</span><span class="z-n">wc</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">CreateWindow</span><span class="z-p">(</span><span class="z-n">class_atom</span><span class="z-p">,</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span><span class="z-p">,</span> <span class="z-kc">None</span><span class="z-p">)</span> <span class="z-k">def</span> <span class="z-nf">_process_message</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">,</span> <span class="z-n">hwnd</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">msg</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">wparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">lparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">):</span> <span class="z-n">WM_CLIPBOARDUPDATE</span> <span class="z-o">=</span> <span class="z-mh">0x031D</span> <span class="z-k">if</span> <span class="z-n">msg</span> <span class="z-o">==</span> <span class="z-n">WM_CLIPBOARDUPDATE</span><span class="z-p">:</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_process_clip</span><span class="z-p">()</span> <span class="z-k">return</span> <span class="z-mi">0</span> <span class="z-k">def</span> <span class="z-nf">_process_clip</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">):</span> <span class="z-n">clip</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">read_clipboard</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-ow">not</span> <span class="z-n">clip</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-k">if</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_update</span><span class="z-p">:</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_update</span><span class="z-p">(</span><span class="z-n">clip</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">clip</span><span class="z-o">.</span><span class="z-n">type</span> <span class="z-o">==</span> <span class="z-s1">&#39;text&#39;</span> <span class="z-ow">and</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_text</span><span class="z-p">:</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_text</span><span class="z-p">(</span><span class="z-n">clip</span><span class="z-o">.</span><span class="z-n">value</span><span class="z-p">)</span> <span class="z-k">elif</span> <span class="z-n">clip</span><span class="z-o">.</span><span class="z-n">type</span> <span class="z-o">==</span> <span class="z-s1">&#39;files&#39;</span> <span class="z-ow">and</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_text</span><span class="z-p">:</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_files</span><span class="z-p">(</span><span class="z-n">clip</span><span class="z-o">.</span><span class="z-n">value</span><span class="z-p">)</span> <span class="z-nd">@staticmethod</span> <span class="z-k">def</span> <span class="z-nf">read_clipboard</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">Optional</span><span class="z-p">[</span><span class="z-n">Clip</span><span class="z-p">]:</span> <span class="z-k">try</span><span class="z-p">:</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">OpenClipboard</span><span class="z-p">()</span> <span class="z-k">def</span> <span class="z-nf">get_formatted</span><span class="z-p">(</span><span class="z-n">fmt</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">IsClipboardFormatAvailable</span><span class="z-p">(</span><span class="z-n">fmt</span><span class="z-p">):</span> <span class="z-k">return</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">GetClipboardData</span><span class="z-p">(</span><span class="z-n">fmt</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-kc">None</span> <span class="z-k">if</span> <span class="z-n">files</span> <span class="z-o">:=</span> <span class="z-n">get_formatted</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_HDROP</span><span class="z-p">):</span> <span class="z-k">return</span> <span class="z-n">Clipboard</span><span class="z-o">.</span><span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;files&#39;</span><span class="z-p">,</span> <span class="z-p">[</span><span class="z-n">Path</span><span class="z-p">(</span><span class="z-n">f</span><span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">f</span> <span class="z-ow">in</span> <span class="z-n">files</span><span class="z-p">])</span> <span class="z-k">elif</span> <span class="z-n">text</span> <span class="z-o">:=</span> <span class="z-n">get_formatted</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_UNICODETEXT</span><span class="z-p">):</span> <span class="z-k">return</span> <span class="z-n">Clipboard</span><span class="z-o">.</span><span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;text&#39;</span><span class="z-p">,</span> <span class="z-n">text</span><span class="z-p">)</span> <span class="z-k">elif</span> <span class="z-n">text_bytes</span> <span class="z-o">:=</span> <span class="z-n">get_formatted</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_TEXT</span><span class="z-p">):</span> <span class="z-k">return</span> <span class="z-n">Clipboard</span><span class="z-o">.</span><span class="z-n">Clip</span><span class="z-p">(</span><span class="z-s1">&#39;text&#39;</span><span class="z-p">,</span> <span class="z-n">text_bytes</span><span class="z-o">.</span><span class="z-n">decode</span><span class="z-p">())</span> <span class="z-k">elif</span> <span class="z-n">bitmap_handle</span> <span class="z-o">:=</span> <span class="z-n">get_formatted</span><span class="z-p">(</span><span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">CF_BITMAP</span><span class="z-p">):</span> <span class="z-c1"># TODO: handle screenshots</span> <span class="z-k">pass</span> <span class="z-k">return</span> <span class="z-kc">None</span> <span class="z-k">finally</span><span class="z-p">:</span> <span class="z-n">win32clipboard</span><span class="z-o">.</span><span class="z-n">CloseClipboard</span><span class="z-p">()</span> <span class="z-k">def</span> <span class="z-nf">listen</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_trigger_at_start</span><span class="z-p">:</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_process_clip</span><span class="z-p">()</span> <span class="z-k">def</span> <span class="z-nf">runner</span><span class="z-p">():</span> <span class="z-n">hwnd</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_create_window</span><span class="z-p">()</span> <span class="z-n">ctypes</span><span class="z-o">.</span><span class="z-n">windll</span><span class="z-o">.</span><span class="z-n">user32</span><span class="z-o">.</span><span class="z-n">AddClipboardFormatListener</span><span class="z-p">(</span><span class="z-n">hwnd</span><span class="z-p">)</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span> <span class="z-n">th</span> <span class="z-o">=</span> <span class="z-n">threading</span><span class="z-o">.</span><span class="z-n">Thread</span><span class="z-p">(</span><span class="z-n">target</span><span class="z-o">=</span><span class="z-n">runner</span><span class="z-p">,</span> <span class="z-n">daemon</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span> <span class="z-k">while</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">is_alive</span><span class="z-p">():</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">(</span><span class="z-mf">0.25</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">clipboard</span> <span class="z-o">=</span> <span class="z-n">Clipboard</span><span class="z-p">(</span><span class="z-n">on_update</span><span class="z-o">=</span><span class="z-nb">print</span><span class="z-p">,</span> <span class="z-n">trigger_at_start</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">)</span> <span class="z-n">clipboard</span><span class="z-o">.</span><span class="z-n">listen</span><span class="z-p">()</span></code></pre> <p>When we run it and copy some text, or some files, it dumps the clipboard content just as we want it.</p> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">Clipboard.Clip(type=&#39;text&#39;, value=&#39;Clipboard&#39;) </span><span class="z-go">Clipboard.Clip(type=&#39;files&#39;, value=[WindowsPath(&#39;C:/Python39/python.exe&#39;)]) </span></code></pre> <p>I haven't managed to retrieve bitmap from the clipboard when taking a screenshot yet, though it shouldn't be too difficult.</p> <p>It should prove useful for the use case where when you take a screenshot, you can save it automatically as PNG, upload it and copy its URL to clipboard, ready for pasting.</p> <h2 id="using-dumpclip-polling-for-changes">Using <strong>dumpclip</strong>: polling for changes</h2> <p>Before I could navigate around Win32 APIs easily, I used higher level APIs provided in C# to listen to the clipboard. On that end, I created a mini utility called <a href="https://abdus.dev/projects/dumpclip/" target="_blank" rel="noopener"><strong>dumpclip</strong></a> that prints the clipboard content to console as JSON or streams clipboard updates.</p> <p>The first version of <strong>dumpclip</strong> had a single function: dumping the clipboard content to console as JSON.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-p">&gt;</span> <span class="z-n">dumpclip</span><span class="z-p">.</span><span class="z-py">v1</span><span class="z-p">.</span><span class="z-py">exe</span> <span class="z-p">{</span><span class="z-s2">&#34;text&#34;</span><span class="z-err">:</span><span class="z-s2">&#34;monitor&#34;</span><span class="z-p">}</span></code></pre> <p>Calling it from Python is quite straightforward using <code>subprocess</code> module. But that also meant polling for changes every second.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">import</span> <span class="z-nn">threading</span> <span class="z-kn">from</span> <span class="z-nn">time</span> <span class="z-kn">import</span> <span class="z-n">sleep</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span> <span class="z-k">def</span> <span class="z-nf">get_clipboard</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-nb">dict</span><span class="z-p">:</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">run</span><span class="z-p">(</span> <span class="z-p">[</span><span class="z-s2">&#34;dumpclip.v1.exe&#34;</span><span class="z-p">],</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">,</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">returncode</span> <span class="z-o">!=</span> <span class="z-mi">0</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-p">{}</span> <span class="z-k">return</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-p">)</span> <span class="z-k">def</span> <span class="z-nf">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-nb">dict</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">])</span> <span class="z-o">-&gt;</span> <span class="z-kc">None</span><span class="z-p">:</span> <span class="z-k">def</span> <span class="z-nf">monitor</span><span class="z-p">():</span> <span class="z-n">old</span> <span class="z-o">=</span> <span class="z-kc">None</span> <span class="z-k">while</span> <span class="z-kc">True</span><span class="z-p">:</span> <span class="z-n">new</span> <span class="z-o">=</span> <span class="z-n">get_clipboard</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-n">old</span> <span class="z-o">!=</span> <span class="z-n">new</span><span class="z-p">:</span> <span class="z-n">on_change</span><span class="z-p">(</span><span class="z-n">new</span><span class="z-p">)</span> <span class="z-n">old</span> <span class="z-o">=</span> <span class="z-n">new</span> <span class="z-n">sleep</span><span class="z-p">(</span><span class="z-mi">1</span><span class="z-p">)</span> <span class="z-n">th</span> <span class="z-o">=</span> <span class="z-n">threading</span><span class="z-o">.</span><span class="z-n">Thread</span><span class="z-p">(</span><span class="z-n">target</span><span class="z-o">=</span><span class="z-n">monitor</span><span class="z-p">)</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s2">&#34;__main__&#34;</span><span class="z-p">:</span> <span class="z-n">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-o">=</span><span class="z-nb">print</span><span class="z-p">)</span></code></pre> <p>It's functional, but we can do better.</p> <h2 id="using-dumpclip-streaming-clipboard-updates">Using <strong>dumpclip</strong>: streaming clipboard updates</h2> <p>The second iteration of <strong>dumpclip</strong> involved using Win32 APIs. I've used <a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/using-the-clipboard#creating-a-clipboard-format-listener" target="_blank" rel="noopener"><code>AddClipboardFormatListener</code></a> to register a callback for clipboard changes in C#, then retrieved &amp; dumped its content as the new content came in.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-p">&gt;</span> <span class="z-n">dumpclip</span><span class="z-p">.</span><span class="z-py">v2</span><span class="z-p">.</span><span class="z-py">exe</span> <span class="z-p">-</span><span class="z-n">-listen</span> <span class="z-p">{</span><span class="z-s2">&#34;text&#34;</span><span class="z-err">:</span><span class="z-s2">&#34;ClipboardChanged&#34;</span><span class="z-p">}</span> <span class="z-p">{</span><span class="z-s2">&#34;text&#34;</span><span class="z-err">:</span><span class="z-s2">&#34;monitor&#34;</span><span class="z-p">}</span> <span class="z-p">{</span><span class="z-s2">&#34;files&#34;</span><span class="z-err">:</span><span class="z-p">[</span><span class="z-s2">&#34;D:\\path\\to\\file.ext&#34;</span><span class="z-p">]}</span> <span class="z-p">...</span></code></pre> <p>This worked much better. I can process its <code>stdout</code> stream, and trigger a callback directly, instead of polling for changes. But <strong>dumpclip</strong> launched in listener mode never terminates. We need to read its stdout in real-time.</p> <p>To stream <code>stdout</code> of a process, we need to launch it with <code>subprocess.Popen</code> and pipe its output to <code>subprocess.PIPE</code>. Then we can read its <code>stdout</code> in a separate thread. Because, the main thread that launches the process will be waiting for the process to terminate (although it never will).</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">import</span> <span class="z-nn">threading</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span> <span class="z-k">def</span> <span class="z-nf">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-nb">dict</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">])</span> <span class="z-o">-&gt;</span> <span class="z-kc">None</span><span class="z-p">:</span> <span class="z-k">def</span> <span class="z-nf">read_stdout</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-p">:</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">Popen</span><span class="z-p">):</span> <span class="z-k">for</span> <span class="z-n">line</span> <span class="z-ow">in</span> <span class="z-nb">iter</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">readline</span><span class="z-p">,</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-n">line</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">():</span> <span class="z-n">payload</span> <span class="z-o">=</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">line</span><span class="z-p">)</span> <span class="z-n">on_change</span><span class="z-p">(</span><span class="z-n">payload</span><span class="z-p">)</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">Popen</span><span class="z-p">(</span> <span class="z-p">[</span><span class="z-s2">&#34;dumpclip.v2.exe&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;--listen&#34;</span><span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">,</span> <span class="z-p">)</span> <span class="z-n">th</span> <span class="z-o">=</span> <span class="z-n">threading</span><span class="z-o">.</span><span class="z-n">Thread</span><span class="z-p">(</span> <span class="z-n">target</span><span class="z-o">=</span><span class="z-n">read_stdout</span><span class="z-p">,</span> <span class="z-n">args</span><span class="z-o">=</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-p">,),</span> <span class="z-p">)</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span> <span class="z-k">try</span><span class="z-p">:</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">wait</span><span class="z-p">()</span> <span class="z-k">except</span> <span class="z-ne">KeyboardInterrupt</span><span class="z-p">:</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">kill</span><span class="z-p">()</span> <span class="z-k">raise</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s2">&#34;__main__&#34;</span><span class="z-p">:</span> <span class="z-n">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-o">=</span><span class="z-nb">print</span><span class="z-p">)</span></code></pre> <p>Because the process doesn't terminate, the thread that consumes its output doesn't stop, either. It keeps processing the output as new content comes in, and idles if there's nothing to consume, as <code>proc.stdout.readline()</code> call is blocking. When the process gets killed, <code>proc.stdout</code> stops blocking and the thread terminates.</p> <p>To prevent blocking interrupt signal and to allow the script and the process terminate, we need to <code>.wait()</code> the subprocess. This allows <code>KeyboardInterrupt</code> to bubble up and terminate the script (and its subprocesses) when we hit <kbd>Ctrl</kbd> + <kbd>C</kbd>.</p> <h2 id="using-dumpclip-async-workflow">Using dumpclip: async workflow</h2> <p>Just for kicks, I wanted to implement the same operation in async. It turned out to be more straightforward to write and consume. One caveat is that you have to create a wrapper async function to use <code>async</code>/<code>await</code> keywords, so I had to add a <code>main</code> function to do that.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">asyncio</span> <span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">from</span> <span class="z-nn">pathlib</span> <span class="z-kn">import</span> <span class="z-n">Path</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">AsyncIterable</span> <span class="z-k">async</span> <span class="z-k">def</span> <span class="z-nf">monitor_clipboard</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">AsyncIterable</span><span class="z-p">[</span><span class="z-nb">dict</span><span class="z-p">]:</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-k">await</span> <span class="z-n">asyncio</span><span class="z-o">.</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">create_subprocess_exec</span><span class="z-p">(</span> <span class="z-s2">&#34;dumpclip.exe&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;--listen&#34;</span><span class="z-p">,</span> <span class="z-n">cwd</span><span class="z-o">=</span><span class="z-nb">str</span><span class="z-p">(</span><span class="z-n">Path</span><span class="z-p">(</span><span class="z-vm">__file__</span><span class="z-p">)</span><span class="z-o">.</span><span class="z-n">parent</span><span class="z-o">.</span><span class="z-n">resolve</span><span class="z-p">()),</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">asyncio</span><span class="z-o">.</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">,</span> <span class="z-p">)</span> <span class="z-k">while</span> <span class="z-kc">True</span><span class="z-p">:</span> <span class="z-k">if</span> <span class="z-n">raw_bytes</span> <span class="z-o">:=</span> <span class="z-k">await</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">readline</span><span class="z-p">():</span> <span class="z-n">text</span> <span class="z-o">=</span> <span class="z-n">raw_bytes</span><span class="z-o">.</span><span class="z-n">decode</span><span class="z-p">()</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-n">text</span><span class="z-p">:</span> <span class="z-k">yield</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">text</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s2">&#34;__main__&#34;</span><span class="z-p">:</span> <span class="z-k">async</span> <span class="z-k">def</span> <span class="z-nf">main</span><span class="z-p">():</span> <span class="z-k">async</span> <span class="z-k">for</span> <span class="z-n">clip</span> <span class="z-ow">in</span> <span class="z-n">monitor_clipboard</span><span class="z-p">():</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-n">clip</span><span class="z-p">)</span> <span class="z-n">asyncio</span><span class="z-o">.</span><span class="z-n">get_event_loop</span><span class="z-p">()</span><span class="z-o">.</span><span class="z-n">run_until_complete</span><span class="z-p">(</span><span class="z-n">main</span><span class="z-p">())</span></code></pre> <p>That's it.<br /> Cheers ✌</p> <p>If you've found this post useful, consider sharing it.</p> Sat, 28 Nov 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/dumpclip/ https://abdus.dev/projects/dumpclip/ <p><img src="/projects/dumpclip/./dumpclip.png" alt="" /></p> <p><strong>dumpclip</strong> is a simple utility that prints clipboard contents to the console as JSON. It supports text and files content. You can find the <a href="https://github.com/abdusco/dumpclip/releases" target="_blank" rel="noopener">latest release</a> on Github:</p> <a class="repo-card" href="https://github.com/abdusco/dumpclip" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>dumpclip</strong> </span> <span class="repo-card__description">A utility for dumping clipboard contents to console</span> <span class="repo-card__meta"> <span>C#</span> <span aria-label="3 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 3 </span> <span aria-label="3 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 3 </span> </span> </a> <h2 id="usage">Usage</h2> <p>Copy some text into the clipboard and run the program.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">dumpclip</span><span class="z-p">.</span><span class="z-py">exe</span></code></pre> <p>Clipboard contents will be serialized as JSON and written to the console:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span><span class="z-nt">&#34;text&#34;</span><span class="z-p">:</span><span class="z-s2">&#34;monitor&#34;</span><span class="z-p">}</span></code></pre> <p>Copy some files and try again.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span><span class="z-nt">&#34;files&#34;</span><span class="z-p">:[</span><span class="z-s2">&#34;D:\\_dev\\web\\abdus.dev\\src\\projects\\askme\\askme.gif&#34;</span><span class="z-p">,</span><span class="z-s2">&#34;D:\\_dev\\web\\abdus.dev\\src\\projects\\askme\\askme.md&#34;</span><span class="z-p">]}</span></code></pre> <h2 id="listen-to-clipboard-changes">Listen to clipboard changes</h2> <p>Run the program with <code>--listen</code> flag.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">dumpclip</span><span class="z-p">.</span><span class="z-py">exe</span> <span class="z-p">-</span><span class="z-n">-listen</span></code></pre> <p>The program will listen to the clipboard and dump contents to the console when the clipboard content changes. You can stop listening by exiting the program with <kbd>Ctrl</kbd> + <kbd>C</kbd></p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span><span class="z-nt">&#34;files&#34;</span><span class="z-p">:[</span><span class="z-s2">&#34;D:\\_dev\\windows\\ClipboardDemo\\DumpClipboard\\bin\\Release\\dumpclip.exe&#34;</span><span class="z-p">]}</span> <span class="z-p">{</span><span class="z-nt">&#34;text&#34;</span><span class="z-p">:</span><span class="z-s2">&#34;System.ValueTuple&#34;</span><span class="z-p">}</span></code></pre> <h3 id="python-script-for-monitoring-clipboard">Python script for monitoring clipboard</h3> <p>The main reason I've built this utility was to integrate it into a Python script. So here's a simple function that runs the app and captures its stdout and calls the given callback.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">import</span> <span class="z-nn">threading</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span> <span class="z-k">def</span> <span class="z-nf">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-nb">dict</span><span class="z-p">],</span> <span class="z-kc">None</span><span class="z-p">])</span> <span class="z-o">-&gt;</span> <span class="z-kc">None</span><span class="z-p">:</span> <span class="z-k">def</span> <span class="z-nf">listen_stdout</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-p">:</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">Popen</span><span class="z-p">):</span> <span class="z-k">for</span> <span class="z-n">line</span> <span class="z-ow">in</span> <span class="z-nb">iter</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">readline</span><span class="z-p">,</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-n">line</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">():</span> <span class="z-n">payload</span> <span class="z-o">=</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">line</span><span class="z-p">)</span> <span class="z-n">on_change</span><span class="z-p">(</span><span class="z-n">payload</span><span class="z-p">)</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">Popen</span><span class="z-p">(</span> <span class="z-p">[</span><span class="z-s2">&#34;dumpclip.exe&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;--listen&#34;</span><span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">universal_newlines</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">,</span> <span class="z-p">)</span> <span class="z-n">th</span> <span class="z-o">=</span> <span class="z-n">threading</span><span class="z-o">.</span><span class="z-n">Thread</span><span class="z-p">(</span> <span class="z-n">target</span><span class="z-o">=</span><span class="z-n">listen_stdout</span><span class="z-p">,</span> <span class="z-n">args</span><span class="z-o">=</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-p">,),</span> <span class="z-p">)</span> <span class="z-n">th</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span> <span class="z-k">try</span><span class="z-p">:</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">wait</span><span class="z-p">()</span> <span class="z-k">except</span> <span class="z-ne">KeyboardInterrupt</span><span class="z-p">:</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">kill</span><span class="z-p">()</span> <span class="z-k">raise</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s2">&#34;__main__&#34;</span><span class="z-p">:</span> <span class="z-n">monitor_clipboard</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-o">=</span><span class="z-nb">print</span><span class="z-p">)</span></code></pre> <p>I've also written <a href="/posts/monitor-clipboard/">a short post</a> as to why I made this and how it progressed.</p> Fri, 27 Nov 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/askme/ https://abdus.dev/projects/askme/ <p><strong>askme</strong> is a mini utility that prompts user questions and prints answers in key-value format to the console. Questions are provided as command arguments as <code>&quot;question&quot;</code> or as <code>&quot;question=default answer&quot;</code> format. You can also specify a key with <code>key:</code> prefix.</p> <p>You can find the <a href="https://github.com/abdusco/askme/releases" target="_blank" rel="noopener">latest release</a> on Github:</p> <a class="repo-card" href="https://github.com/abdusco/askme" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdusco</span> <span aria-hidden="true">/</span> <strong>askme</strong> </span> <span class="repo-card__description">A mini tool to prompt questions in a dialog and print the answers to console.</span> <span class="repo-card__meta"> <span>C#</span> <span aria-label="1 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 1 </span> <span aria-label="1 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 1 </span> </span> </a> <h2 id="usage">Usage</h2> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">askme</span><span class="z-p">.</span><span class="z-py">exe</span> <span class="z-s2">&#34;filename:Filename for the export=export.zip&#34;</span> <span class="z-s2">&#34;author:Author?&#34;</span> <span class="z-n">year</span></code></pre> <p>A dialog should appear with inputs to type the answers:</p> <p><video src="./askme.webm" autoplay loop controls playsinline></video></p> <p>When saved by clicking the Save button or hitting <kbd>ENTER</kbd> key, the app exits, and the user inputs are printed to console as JSON.</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">{</span><span class="z-nt">&#34;filename&#34;</span><span class="z-p">:</span><span class="z-s2">&#34;report.zip&#34;</span><span class="z-p">,</span><span class="z-nt">&#34;author&#34;</span><span class="z-p">:</span><span class="z-s2">&#34;abdus&#34;</span><span class="z-p">,</span><span class="z-nt">&#34;year&#34;</span><span class="z-p">:</span><span class="z-s2">&#34;2020&#34;</span><span class="z-p">}</span></code></pre> <p>The response can then be consumed from any script by reading the stdout and deserializing the JSON.</p> <p>Hitting <kbd>Esc</kbd> key closes the window and nothing is printed to console.</p> <h2 id="return-codes">Return codes</h2> <table> <thead> <tr> <th align="right">Code</th> <th align="left">Reason</th> </tr> </thead> <tbody> <tr> <td align="right"><code>0</code></td> <td align="left">User submitted the answers</td> </tr> <tr> <td align="right"><code>1</code></td> <td align="left">Failed to parse questions</td> </tr> <tr> <td align="right"><code>2</code></td> <td align="left">User cancelled the prompt</td> </tr> </tbody> </table> Sat, 14 Nov 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/getting-public-url-behind-reverse-proxy/ https://abdus.dev/posts/getting-public-url-behind-reverse-proxy/ <p>I'm building a small library for broadcasting URLs to a service that pings these URLs at regular intervals. The service works like crontab on steroids. It keeps a list of URLs and when those URLs should be called using cron expressions.</p> <p>Each URL is handled by a handler that runs a job in the background. This lets me perform routine tasks without having to run a separate worker process. This approach has its faults, but for short &amp; light tasks, it's more than enough.</p> <p>At startup, I gather a list of cron jobs inside a <code>HostedService</code>, then broadcast them to crontab-service to have them pinged later.</p> <h2 id="problem-reverse-proxies">Problem: reverse proxies</h2> <p>A problem I faced while building the URLs was to find out the public-facing URL for the web app. I can't just get these URLs from Kestrel or any other component because my app will be listening to localhost at some port. App has no idea about whether there's a server proxying requests to it, such as IIS, Caddy, Nginx, etc. It has to wait until the first request to find out the actual URL that reaches the app.</p> <p>A workaround I've implemented was to supply a URL template <code>https://public.url/-/cronjobs/{name}</code> and generate URLs with it. But if the URL is out of my control, as in the case of a lambda function with a random URL assigned by a cloud provider, I would still have to get the URL.</p> <p>When working inside an HTTP context, I can find out which URL is being handled using</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kt">var</span> <span class="z-n">url</span> <span class="z-p">=</span> <span class="z-n">context</span><span class="z-p">.</span><span class="z-n">Request</span><span class="z-p">.</span><span class="z-n">GetDisplayUrl</span><span class="z-p">()</span></code></pre> <p>But a background service works outside the HTTP context, so it still cannot know about any HTTP event.</p> <h2 id="solution">Solution</h2> <p>The solution I implemented was to create a singleton to store the URL, and put a middleware at the beginning of the request pipeline. When the first request arrives, I trigger an event, and the worker waiting for this event continues execution.</p> <p>The singleton service is a wrapper around <a href="https://docs.microsoft.com/en-us/dotnet/api/system.threading.manualresetevent" target="_blank" rel="noopener"><code>ManualResetEventSlim</code></a>.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">PublicUrlRegistry</span> <span class="z-p">{</span> <span class="z-kd">private</span> <span class="z-kd">static</span> <span class="z-k">readonly</span> <span class="z-n">Lazy</span><span class="z-p">&lt;</span><span class="z-n">ManualResetEventSlim</span><span class="z-p">&gt;</span> <span class="z-n">@event</span> <span class="z-p">=</span> <span class="z-k">new</span> <span class="z-n">Lazy</span><span class="z-p">&lt;</span><span class="z-n">ManualResetEventSlim</span><span class="z-p">&gt;();</span> <span class="z-kd">private</span> <span class="z-kd">static</span> <span class="z-n">Uri</span> <span class="z-n">_url</span><span class="z-p">;</span> <span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">Set</span><span class="z-p">(</span><span class="z-n">Uri</span> <span class="z-n">url</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_url</span> <span class="z-p">=</span> <span class="z-n">url</span><span class="z-p">;</span> <span class="z-n">@event</span><span class="z-p">.</span><span class="z-n">Value</span><span class="z-p">.</span><span class="z-n">Set</span><span class="z-p">();</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-kt">bool</span> <span class="z-n">IsSet</span> <span class="z-p">=&gt;</span> <span class="z-n">@event</span><span class="z-p">.</span><span class="z-n">Value</span><span class="z-p">.</span><span class="z-n">IsSet</span><span class="z-p">;</span> <span class="z-kd">public</span> <span class="z-n">Uri</span> <span class="z-n">WaitForUrl</span><span class="z-p">(</span><span class="z-n">CancellationToken</span> <span class="z-n">cancellationToken</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">@event</span><span class="z-p">.</span><span class="z-n">Value</span><span class="z-p">.</span><span class="z-n">Wait</span><span class="z-p">(</span><span class="z-n">cancellationToken</span><span class="z-p">);</span> <span class="z-k">return</span> <span class="z-n">_url</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>The middleware retrieves and stores the URL of the request.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"> <span class="z-kd">internal</span> <span class="z-kd">static</span> <span class="z-k">class</span> <span class="z-nc">ApplicationBuilderExtensions</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-kd">static</span> <span class="z-n">IApplicationBuilder</span> <span class="z-n">UsePublicUrlDetector</span><span class="z-p">(</span><span class="z-k">this</span> <span class="z-n">IApplicationBuilder</span> <span class="z-n">app</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-n">app</span><span class="z-p">.</span><span class="z-n">Use</span><span class="z-p">((</span><span class="z-n">context</span><span class="z-p">,</span> <span class="z-n">next</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-kt">var</span> <span class="z-n">registry</span> <span class="z-p">=</span> <span class="z-n">context</span><span class="z-p">.</span><span class="z-n">RequestServices</span><span class="z-p">.</span><span class="z-n">GetRequiredService</span><span class="z-p">&lt;</span><span class="z-n">PublicUrlRegistry</span><span class="z-p">&gt;();</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-n">registry</span><span class="z-p">.</span><span class="z-n">IsSet</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-k">return</span> <span class="z-n">next</span><span class="z-p">();</span> <span class="z-p">}</span> <span class="z-n">registry</span><span class="z-p">.</span><span class="z-n">Set</span><span class="z-p">(</span><span class="z-k">new</span> <span class="z-n">Uri</span><span class="z-p">(</span><span class="z-n">context</span><span class="z-p">.</span><span class="z-n">Request</span><span class="z-p">.</span><span class="z-n">GetDisplayUrl</span><span class="z-p">()));</span> <span class="z-k">return</span> <span class="z-n">next</span><span class="z-p">();</span> <span class="z-p">});</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>Background service waits until the URL is set, then continues execution.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">BackgroundWorker</span> <span class="z-p">:</span> <span class="z-n">BackgroundService</span> <span class="z-p">{</span> <span class="z-kd">private</span> <span class="z-k">readonly</span> <span class="z-n">ILogger</span><span class="z-p">&lt;</span><span class="z-n">BackgroundWorker</span><span class="z-p">&gt;</span> <span class="z-n">_logger</span><span class="z-p">;</span> <span class="z-kd">private</span> <span class="z-k">readonly</span> <span class="z-n">PublicUrlRegistry</span> <span class="z-n">_registry</span><span class="z-p">;</span> <span class="z-kd">public</span> <span class="z-n">BackgroundWorker</span><span class="z-p">(</span><span class="z-n">ILogger</span><span class="z-p">&lt;</span><span class="z-n">BackgroundWorker</span><span class="z-p">&gt;</span> <span class="z-n">logger</span><span class="z-p">,</span> <span class="z-n">PublicUrlRegistry</span> <span class="z-n">registry</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span> <span class="z-p">=</span> <span class="z-n">logger</span><span class="z-p">;</span> <span class="z-n">_registry</span> <span class="z-p">=</span> <span class="z-n">registry</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-kd">protected</span> <span class="z-kd">override</span> <span class="z-n">Task</span> <span class="z-n">ExecuteAsync</span><span class="z-p">(</span><span class="z-n">CancellationToken</span> <span class="z-n">stoppingToken</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">_logger</span><span class="z-p">.</span><span class="z-n">LogInformation</span><span class="z-p">(</span><span class="z-s">&#34;Waiting for the first request&#34;</span><span class="z-p">);</span> <span class="z-k">try</span> <span class="z-p">{</span> <span class="z-c1">// this line is blocking execution until event is set</span> <span class="z-kt">var</span> <span class="z-n">url</span> <span class="z-p">=</span> <span class="z-n">_registry</span><span class="z-p">.</span><span class="z-n">WaitForUrl</span><span class="z-p">(</span><span class="z-n">stoppingToken</span><span class="z-p">);</span> <span class="z-n">_logger</span><span class="z-p">.</span><span class="z-n">LogInformation</span><span class="z-p">(</span><span class="z-s">&#34;Got {url}&#34;</span><span class="z-p">,</span> <span class="z-n">url</span><span class="z-p">.</span><span class="z-n">GetLeftPart</span><span class="z-p">(</span><span class="z-n">UriPartial</span><span class="z-p">.</span><span class="z-n">Authority</span><span class="z-p">));</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">(</span><span class="z-n">OperationCanceledException</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// App is stopping</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-n">Task</span><span class="z-p">.</span><span class="z-n">CompletedTask</span><span class="z-p">;</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <p>and configured in <code>Startup</code> class. We need to register service as singleton, and put middleware at the beginning of the request pipeline. Otherwise, other middlewares could short circuit and requests may never reach our middleware.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">Startup</span> <span class="z-p">{</span> <span class="z-c1">// ...</span> <span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">ConfigureServices</span><span class="z-p">(</span><span class="z-n">IServiceCollection</span> <span class="z-n">services</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// not registering hosted service here! read on.</span> <span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddSingleton</span><span class="z-p">&lt;</span><span class="z-n">PublicUrlRegistry</span><span class="z-p">&gt;();</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">Configure</span><span class="z-p">(</span><span class="z-n">IApplicationBuilder</span> <span class="z-n">app</span><span class="z-p">,</span> <span class="z-n">IWebHostEnvironment</span> <span class="z-n">env</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-c1">// middleware comes first!</span> <span class="z-n">app</span><span class="z-p">.</span><span class="z-n">UsePublicUrlDetector</span><span class="z-p">();</span> <span class="z-n">app</span><span class="z-p">.</span><span class="z-n">UseRouting</span><span class="z-p">();</span> <span class="z-n">app</span><span class="z-p">.</span><span class="z-n">UseEndpoints</span><span class="z-p">(</span><span class="z-n">endpoints</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-n">endpoints</span><span class="z-p">.</span><span class="z-n">MapGet</span><span class="z-p">(</span><span class="z-s">&#34;/&#34;</span><span class="z-p">,</span> <span class="z-n">context</span> <span class="z-p">=&gt;</span> <span class="z-n">context</span><span class="z-p">.</span><span class="z-n">Response</span><span class="z-p">.</span><span class="z-n">WriteAsync</span><span class="z-p">(</span><span class="z-s">&#34;hello&#34;</span><span class="z-p">));</span> <span class="z-p">});</span> <span class="z-p">}</span> <span class="z-p">}</span></code></pre> <h2 id="deadlock">Deadlock</h2> <p>So was the plan. But after running the app I realized that the app never starts listening for requests. Because it waits until <code>ExecuteAsync</code> method of the hosted service returns. But that never happens because service waits until the first request.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-c1">// Startup.cs</span> <span class="z-kd">public</span> <span class="z-k">void</span> <span class="z-n">ConfigureServices</span><span class="z-p">(</span><span class="z-n">IServiceCollection</span> <span class="z-n">services</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddHostedService</span><span class="z-p">&lt;</span><span class="z-n">BackgroundWorker</span><span class="z-p">&gt;();</span> <span class="z-c1">// &lt;-- won&#39;t work!</span> <span class="z-n">services</span><span class="z-p">.</span><span class="z-n">AddSingleton</span><span class="z-p">&lt;</span><span class="z-n">PublicUrlRegistry</span><span class="z-p">&gt;();</span> <span class="z-p">}</span></code></pre> <p>We've got a <a href="https://en.wikipedia.org/wiki/Deadlock" target="_blank" rel="noopener">deadlock</a>.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">dotnet</span> <span class="z-n">run</span> <span class="z-n">Building</span><span class="z-p">...</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">WaitForFirstRequestDemo</span><span class="z-p">.</span><span class="z-n">BackgroundWorker</span><span class="z-p">[</span><span class="z-mf">0</span><span class="z-p">]</span> <span class="z-n">Waiting</span> <span class="z-k">for</span> <span class="z-n">the</span> <span class="z-n">first</span> <span class="z-n">request</span></code></pre> <h2 id="making-sure-hosted-service-starts-after-the-server-starts-listening">Making sure hosted service starts after the server starts listening</h2> <p>The solution comes from <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&amp;tabs=visual-studio#ihostedservice-interface" target="_blank" rel="noopener">IHostedService's docs</a>. If we register hosted service in Startup class then it runs before the app starts listening to requests.</p> <p>We need to move the registration to where <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-5.0" target="_blank" rel="noopener"><code>IHost</code></a> is built, i.e. <code>Program</code> class, instead. We need to call <code>ConfigureServices</code> after <code>ConfigureWebHostDefaults</code> and register the service there.</p> <pre data-lang="c#" data-highlighted="true" class="language-c# z-code z-chroma"><code class="language-c#" data-lang="c#"><span class="z-kd">public</span> <span class="z-k">class</span> <span class="z-nc">Program</span> <span class="z-p">{</span> <span class="z-kd">public</span> <span class="z-kd">static</span> <span class="z-k">void</span> <span class="z-n">Main</span><span class="z-p">(</span><span class="z-kt">string</span><span class="z-p">[]</span> <span class="z-n">args</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">CreateHostBuilder</span><span class="z-p">(</span><span class="z-n">args</span><span class="z-p">).</span><span class="z-n">Build</span><span class="z-p">().</span><span class="z-n">Run</span><span class="z-p">();</span> <span class="z-p">}</span> <span class="z-kd">public</span> <span class="z-kd">static</span> <span class="z-n">IHostBuilder</span> <span class="z-n">CreateHostBuilder</span><span class="z-p">(</span><span class="z-kt">string</span><span class="z-p">[]</span> <span class="z-n">args</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-n">Host</span><span class="z-p">.</span><span class="z-n">CreateDefaultBuilder</span><span class="z-p">(</span><span class="z-n">args</span><span class="z-p">)</span> <span class="z-p">.</span><span class="z-n">ConfigureWebHostDefaults</span><span class="z-p">(</span><span class="z-n">webBuilder</span> <span class="z-p">=&gt;</span> <span class="z-p">{</span> <span class="z-n">webBuilder</span><span class="z-p">.</span><span class="z-n">UseStartup</span><span class="z-p">&lt;</span><span class="z-n">Startup</span><span class="z-p">&gt;();</span> <span class="z-p">})</span> <span class="z-c1">// &lt;-- first configure web</span> <span class="z-p">.</span><span class="z-n">ConfigureServices</span><span class="z-p">(</span><span class="z-n">collection</span> <span class="z-p">=&gt;</span> <span class="z-n">collection</span><span class="z-p">.</span><span class="z-n">AddHostedService</span><span class="z-p">&lt;</span><span class="z-n">BackgroundWorker</span><span class="z-p">&gt;());</span> <span class="z-c1">// &lt;-- then register hosted service</span> <span class="z-p">}</span></code></pre> <p>When I run the app and send a request, everything works as it should.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">dotnet</span> <span class="z-n">run</span> <span class="z-n">Building</span><span class="z-p">...</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">Microsoft</span><span class="z-p">.</span><span class="z-py">Hosting</span><span class="z-p">.</span><span class="z-n">Lifetime</span><span class="z-p">[</span><span class="z-mf">0</span><span class="z-p">]</span> <span class="z-n">Now</span> <span class="z-n">listening</span> <span class="z-n">on</span><span class="z-err">:</span> <span class="z-n">https</span><span class="z-err">:</span><span class="z-p">//</span><span class="z-n">localhost</span><span class="z-err">:</span><span class="z-mf">5001</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">Microsoft</span><span class="z-p">.</span><span class="z-py">Hosting</span><span class="z-p">.</span><span class="z-n">Lifetime</span><span class="z-p">[</span><span class="z-mf">0</span><span class="z-p">]</span> <span class="z-n">Now</span> <span class="z-n">listening</span> <span class="z-n">on</span><span class="z-err">:</span> <span class="z-n">http</span><span class="z-err">:</span><span class="z-p">//</span><span class="z-n">localhost</span><span class="z-err">:</span><span class="z-mf">5000</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">WaitForFirstRequestDemo</span><span class="z-p">.</span><span class="z-n">BackgroundWorker</span><span class="z-p">[</span><span class="z-mf">0</span><span class="z-p">]</span> <span class="z-n">Waiting</span> <span class="z-k">for</span> <span class="z-n">the</span> <span class="z-n">first</span> <span class="z-n">request</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">Microsoft</span><span class="z-p">.</span><span class="z-py">AspNetCore</span><span class="z-p">.</span><span class="z-py">Hosting</span><span class="z-p">.</span><span class="z-n">Diagnostics</span><span class="z-p">[</span><span class="z-mf">1</span><span class="z-p">]</span> <span class="z-n">Request</span> <span class="z-n">starting</span> <span class="z-n">HTTP</span><span class="z-p">/</span><span class="z-mf">1.1</span> <span class="z-n">GET</span> <span class="z-n">https</span><span class="z-err">:</span><span class="z-p">//</span><span class="z-n">localhost</span><span class="z-err">:</span><span class="z-mf">5001</span><span class="z-p">/</span> <span class="z-p">-</span> <span class="z-p">-</span> <span class="z-n">info</span><span class="z-err">:</span> <span class="z-n">WaitForFirstRequestDemo</span><span class="z-p">.</span><span class="z-n">BackgroundWorker</span><span class="z-p">[</span><span class="z-mf">0</span><span class="z-p">]</span> <span class="z-n">Got</span> <span class="z-n">https</span><span class="z-err">:</span><span class="z-p">//</span><span class="z-n">localhost</span><span class="z-err">:</span><span class="z-mf">5001</span></code></pre> Tue, 27 Oct 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/update-path-powershell/ https://abdus.dev/posts/update-path-powershell/ <p>I am writing a script to configure a fresh Windows installation to fit my workflow. This usually involves installing <a href="https://chocolatey.org/install" target="_blank" rel="noopener">chocolatey</a> as my package manager and installing programs I need. Along with it, I also need to restore settings for certain apps and update the <a href="https://en.wikipedia.org/wiki/PATH_(variable)" target="_blank" rel="noopener"><code>PATH</code></a> environment variable.</p> <p>Until today I've always added paths manually, but it turns out, there's an easier way. <a href="https://docs.microsoft.com/en-us/dotnet/api/system.environment.setenvironmentvariable" target="_blank" rel="noopener">Thanks to some googling</a>, I was able to create this script that adds the given dir to <code>PATH</code> environment variable for the current user.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-kd">function</span><span class="z-w"> </span><span class="z-nb">Add-ToUserPath</span> <span class="z-p">{</span> <span class="z-k">param</span> <span class="z-p">(</span> <span class="z-p">[</span><span class="z-nb">Parameter</span><span class="z-p">(</span><span class="z-na">Mandatory</span><span class="z-p">=</span><span class="z-vm">$true</span><span class="z-p">)]</span> <span class="z-p">[</span><span class="z-nb">ValidateNotNullOrEmpty</span><span class="z-p">()]</span> <span class="z-p">[</span><span class="z-no">string</span><span class="z-p">]</span> <span class="z-nv">$dir</span> <span class="z-p">)</span> <span class="z-nv">$dir</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-nb">Resolve-Path</span> <span class="z-nv">$dir</span><span class="z-p">)</span> <span class="z-nv">$path</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">Environment</span><span class="z-p">]::</span><span class="z-n">GetEnvironmentVariable</span><span class="z-p">(</span><span class="z-s2">&#34;PATH&#34;</span><span class="z-p">,</span> <span class="z-p">[</span><span class="z-no">System.EnvironmentVariableTarget</span><span class="z-p">]::</span><span class="z-n">User</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-p">(!(</span><span class="z-nv">$path</span><span class="z-p">.</span><span class="z-py">Contains</span><span class="z-p">(</span><span class="z-nv">$dir</span><span class="z-p">)))</span> <span class="z-p">{</span> <span class="z-c"># backup the current value</span> <span class="z-s2">&#34;PATH=</span><span class="z-nv">$path</span><span class="z-s2">&#34;</span> <span class="z-p">|</span> <span class="z-nb">Set-Content</span> <span class="z-n">-Path</span> <span class="z-s2">&#34;</span><span class="z-nv">$env:USERPROFILE</span><span class="z-s2">/path.env&#34;</span> <span class="z-c"># append dir to path</span> <span class="z-p">[</span><span class="z-no">Environment</span><span class="z-p">]::</span><span class="z-n">SetEnvironmentVariable</span><span class="z-p">(</span><span class="z-s2">&#34;PATH&#34;</span><span class="z-p">,</span> <span class="z-nv">$path</span> <span class="z-p">+</span> <span class="z-s2">&#34;;</span><span class="z-nv">$dir</span><span class="z-s2">&#34;</span><span class="z-p">,</span> <span class="z-p">[</span><span class="z-no">EnvironmentVariableTarget</span><span class="z-p">]::</span><span class="z-n">User</span><span class="z-p">)</span> <span class="z-nb">Write-Host</span> <span class="z-s2">&#34;Added </span><span class="z-nv">$dir</span><span class="z-s2"> to PATH&#34;</span> <span class="z-k">return</span> <span class="z-p">}</span> <span class="z-nb">Write-Error</span> <span class="z-s2">&#34;</span><span class="z-nv">$dir</span><span class="z-s2"> is already in PATH&#34;</span> <span class="z-p">}</span></code></pre> <p>It expects a path to a directory as the argument:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nb">Add-ToUserPath</span> <span class="z-s2">&#34;</span><span class="z-nv">$env:USERPROFILE</span><span class="z-s2">/.bin&#34;</span></code></pre> <p>If you want to set the environment variable for all users, change the target <code>[System.EnvironmentVariableTarget]</code> parameter from <code>User</code> to <code>Machine</code>.</p> <h2 id="adding-the-script-to-powershell-profile">Adding the script to PowerShell profile</h2> <p>Pasting and running this code, or importing a file every time we need to update <code>PATH</code> defeats the purpose. We need a way to access it quickly. That's where the PowerShell <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_profiles" target="_blank" rel="noopener">profile</a> comes into play. It's a file (similar to <code>.bashrc</code> in bash) that runs everytime a PowerShell session starts<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p> <p>Open the profile with an editor.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">notepad</span> <span class="z-nv">$PROFILE</span></code></pre> <p>Paste the function into this file and save it. When you open a new powershell window, you should be able to use the function.</p> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>Unless <code>-NoProfile</code> argument is given&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Sun, 02 Aug 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/fixing-wsl2-localhost-access-issue/ https://abdus.dev/posts/fixing-wsl2-localhost-access-issue/ <p>I've been using <a href="https://docs.microsoft.com/en-us/windows/wsl/" target="_blank" rel="noopener">WSL</a> ever since it came out. It was nice being able to run Linux on Windows without the overhead of VMs. But it had some issues, like not being able to run Docker natively. Microsoft then released a new version called WSL2 which has brought native Docker support.</p> <p>One feature of WSL is that it allows sharing IP address space for services listening to <a href="https://docs.microsoft.com/en-us/windows/wsl/faq#how-do-i-access-a-port-from-wsl-in-windows" target="_blank" rel="noopener">localhost</a>. This means one can access servers running on WSL from Windows as if it were running on Windows. This lets us, for instance, access a Docker container listening to <code>0.0.0.0:8000</code> on WSL from Windows using <code>localhost:8000</code>.</p> <h2 id="problem">Problem</h2> <p>Localhost redirection <a href="https://github.com/microsoft/WSL/issues?q=is%3Aissue+is%3Aopen+localhost" target="_blank" rel="noopener">often fails</a> for some reason, such as when PC sleeps and wakes up, and localhost access to Linux services does not work anymore.</p> <h2 id="solution">Solution</h2> <p>We can get the internal IP a WSL instance (in my case an Ubuntu distro) has using</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">$ ip -4 addr show eth0 5: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu <span class="z-m">1500</span> qdisc mq state UP group default qlen <span class="z-m">1000</span> inet 172.27.106.75/20 brd 172.27.111.255 scope global eth0 valid_lft forever preferred_lft forever</code></pre> <p>We can see the IP address <code>172.27.106.75</code> which we can ping from Windows just fine.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">ping</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">106</span><span class="z-p">.</span><span class="z-py">75</span> <span class="z-n">Pinging</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">106</span><span class="z-p">.</span><span class="z-py">75</span> <span class="z-n">with</span> <span class="z-mf">32</span> <span class="z-n">bytes</span> <span class="z-n">of</span> <span class="z-n">data</span><span class="z-err">:</span> <span class="z-n">Reply</span> <span class="z-n">from</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">106</span><span class="z-p">.</span><span class="z-mf">75</span><span class="z-err">:</span> <span class="z-n">bytes</span><span class="z-p">=</span><span class="z-mf">32</span> <span class="z-n">time</span><span class="z-p">&lt;</span><span class="z-n">1ms</span> <span class="z-n">TTL</span><span class="z-p">=</span><span class="z-mf">64</span></code></pre> <p>We can run a Python server and reach it from Windows.</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">$ python3 -m http.server Serving HTTP on 0.0.0.0 port <span class="z-m">8000</span> <span class="z-o">(</span>http://0.0.0.0:8000/<span class="z-o">)</span> ...</code></pre> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-nb">curl </span><span class="z-n">-I</span> <span class="z-n">http</span><span class="z-err">:</span><span class="z-p">//</span><span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">106</span><span class="z-p">.</span><span class="z-mf">75</span><span class="z-err">:</span><span class="z-mf">8000</span> <span class="z-n">HTTP</span><span class="z-p">/</span><span class="z-mf">1.0</span> <span class="z-mf">200</span> <span class="z-n">OK</span> <span class="z-p">...</span></code></pre> <p>Now we can make the experience better by not having to type in the full IP address everytime we want to access WSL. We can use the <a href="https://en.wikipedia.org/wiki/Hosts_(file)" target="_blank" rel="noopener"><code>hosts</code> file</a> for that. But there's a catch: we can't just use <code>localhost</code>, because that'd break a lot of systems that rely on <code>localhost</code> working as a loopback address. That's why I chose <code>wsl</code> as the hostname, you're free to use something else.</p> <p>For this I've written a PowerShell script that gets the IP address of WSL instance, then adds (or updates) an entry in the <code>hosts</code> file.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nv">$hostname</span> <span class="z-p">=</span> <span class="z-s2">&#34;wsl&#34;</span> <span class="z-c"># find ip of eth0</span> <span class="z-nv">$ifconfig</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-n">wsl</span> <span class="z-p">--</span> <span class="z-n">ip</span> <span class="z-mf">-4</span> <span class="z-n">addr</span> <span class="z-n">show</span> <span class="z-n">eth0</span><span class="z-p">)</span> <span class="z-nv">$ipPattern</span> <span class="z-p">=</span> <span class="z-s2">&#34;((\d+\.?){4})&#34;</span> <span class="z-nv">$ip</span> <span class="z-p">=</span> <span class="z-p">([</span><span class="z-no">regex</span><span class="z-p">]</span><span class="z-s2">&#34;inet </span><span class="z-nv">$ipPattern</span><span class="z-s2">&#34;</span><span class="z-p">).</span><span class="z-py">Match</span><span class="z-p">(</span><span class="z-nv">$ifconfig</span><span class="z-p">).</span><span class="z-n">Groups</span><span class="z-p">[</span><span class="z-mf">1</span><span class="z-p">].</span><span class="z-py">Value</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-o">-not</span> <span class="z-nv">$ip</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-n">exit</span> <span class="z-p">}</span> <span class="z-nb">Write-Host</span> <span class="z-nv">$ip</span> <span class="z-nv">$hostsPath</span> <span class="z-p">=</span> <span class="z-s2">&#34;</span><span class="z-nv">$env:windir</span><span class="z-s2">/system32/drivers/etc/hosts&#34;</span> <span class="z-nv">$hosts</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-nb">Get-Content</span> <span class="z-n">-Path</span> <span class="z-nv">$hostsPath</span> <span class="z-n">-Raw</span> <span class="z-n">-ErrorAction</span> <span class="z-n">Ignore</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-vm">$null</span> <span class="z-o">-eq</span> <span class="z-nv">$hosts</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nv">$hosts</span> <span class="z-p">=</span> <span class="z-s2">&#34;&#34;</span> <span class="z-p">}</span> <span class="z-nv">$hosts</span> <span class="z-p">=</span> <span class="z-nv">$hosts</span><span class="z-p">.</span><span class="z-py">Trim</span><span class="z-p">()</span> <span class="z-c"># update or add wsl ip</span> <span class="z-nv">$find</span> <span class="z-p">=</span> <span class="z-s2">&#34;</span><span class="z-nv">$ipPattern</span><span class="z-s2">\s+</span><span class="z-nv">$hostname</span><span class="z-s2">&#34;</span> <span class="z-nv">$entry</span> <span class="z-p">=</span> <span class="z-s2">&#34;</span><span class="z-nv">$ip</span><span class="z-s2"> </span><span class="z-nv">$hostname</span><span class="z-s2">&#34;</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-nv">$hosts</span> <span class="z-o">-match</span> <span class="z-nv">$find</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nv">$hosts</span> <span class="z-p">=</span> <span class="z-nv">$hosts</span> <span class="z-o">-replace</span> <span class="z-nv">$find</span><span class="z-p">,</span> <span class="z-nv">$entry</span> <span class="z-p">}</span> <span class="z-k">else</span> <span class="z-p">{</span> <span class="z-nv">$hosts</span> <span class="z-p">=</span> <span class="z-s2">&#34;</span><span class="z-nv">$hosts</span><span class="z-se">`n</span><span class="z-nv">$entry</span><span class="z-s2">&#34;</span><span class="z-p">.</span><span class="z-py">Trim</span><span class="z-p">()</span> <span class="z-p">}</span> <span class="z-k">try</span> <span class="z-p">{</span> <span class="z-nv">$temp</span> <span class="z-p">=</span> <span class="z-s2">&#34;</span><span class="z-nv">$hostsPath</span><span class="z-s2">.new&#34;</span> <span class="z-nb">New-Item</span> <span class="z-n">-Path</span> <span class="z-nv">$temp</span> <span class="z-n">-ItemType</span> <span class="z-n">File</span> <span class="z-n">-Force</span> <span class="z-p">|</span> <span class="z-nb">Out-Null</span> <span class="z-nb">Set-Content</span> <span class="z-n">-Path</span> <span class="z-nv">$temp</span> <span class="z-nv">$hosts</span> <span class="z-nb">Move-Item</span> <span class="z-n">-Path</span> <span class="z-nv">$temp</span> <span class="z-n">-Destination</span> <span class="z-nv">$hostsPath</span> <span class="z-n">-Force</span> <span class="z-p">}</span> <span class="z-k">catch</span> <span class="z-p">{</span> <span class="z-nb">Write-Error</span> <span class="z-s2">&#34;cannot update wsl ip&#34;</span> <span class="z-p">}</span></code></pre> <p>Save this script as <code>wsl.ps1</code> and run it as admin to add an entry for WSL in hosts file.</p> <pre data-lang="cmd" data-highlighted="true" class="language-cmd z-code z-chroma"><code class="language-cmd" data-lang="cmd">powershell -file wsl.ps1</code></pre> <h2 id="automating-ip-renewal">Automating IP renewal</h2> <p>Everytime a WSL instance restarts, it gets a new IP address, so we need to keep the hosts file updated.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">wsl</span> <span class="z-p">--</span> <span class="z-n">ip</span> <span class="z-mf">-4</span> <span class="z-n">addr</span> <span class="z-n">show</span> <span class="z-n">eth0</span> <span class="z-mf">5</span><span class="z-err">:</span> <span class="z-n">eth0</span><span class="z-err">:</span> <span class="z-p">&lt;</span><span class="z-n">BROADCAST</span><span class="z-p">,</span><span class="z-n">MULTICAST</span><span class="z-p">,</span><span class="z-n">UP</span><span class="z-p">,</span><span class="z-n">LOWER_UP</span><span class="z-p">&gt;</span> <span class="z-n">mtu</span> <span class="z-mf">1500</span> <span class="z-n">qdisc</span> <span class="z-n">mq</span> <span class="z-n">state</span> <span class="z-n">UP</span> <span class="z-nb">group </span><span class="z-k">default</span> <span class="z-n">qlen</span> <span class="z-mf">1000</span> <span class="z-n">inet</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">101</span><span class="z-p">.</span><span class="z-mf">9</span><span class="z-p">/</span><span class="z-mf">20</span> <span class="z-n">brd</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">111</span><span class="z-p">.</span><span class="z-py">255</span> <span class="z-n">scope</span> <span class="z-n">global</span> <span class="z-n">eth0</span> <span class="z-n">valid_lft</span> <span class="z-n">forever</span> <span class="z-n">preferred_lft</span> <span class="z-n">forever</span> <span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">wsl</span> <span class="z-p">-</span><span class="z-n">-shutdown</span> <span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">wsl</span> <span class="z-p">--</span> <span class="z-n">ip</span> <span class="z-mf">-4</span> <span class="z-n">addr</span> <span class="z-n">show</span> <span class="z-n">eth0</span> <span class="z-mf">5</span><span class="z-err">:</span> <span class="z-n">eth0</span><span class="z-err">:</span> <span class="z-p">&lt;</span><span class="z-n">BROADCAST</span><span class="z-p">,</span><span class="z-n">MULTICAST</span><span class="z-p">,</span><span class="z-n">UP</span><span class="z-p">,</span><span class="z-n">LOWER_UP</span><span class="z-p">&gt;</span> <span class="z-n">mtu</span> <span class="z-mf">1500</span> <span class="z-n">qdisc</span> <span class="z-n">mq</span> <span class="z-n">state</span> <span class="z-n">UP</span> <span class="z-nb">group </span><span class="z-k">default</span> <span class="z-n">qlen</span> <span class="z-mf">1000</span> <span class="z-n">inet</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">102</span><span class="z-p">.</span><span class="z-mf">184</span><span class="z-p">/</span><span class="z-mf">20</span> <span class="z-n">brd</span> <span class="z-mf">172.27</span><span class="z-p">.</span><span class="z-py">111</span><span class="z-p">.</span><span class="z-py">255</span> <span class="z-n">scope</span> <span class="z-n">global</span> <span class="z-n">eth0</span> <span class="z-n">valid_lft</span> <span class="z-n">forever</span> <span class="z-n">preferred_lft</span> <span class="z-n">forever</span></code></pre> <p>We can use Scheduled Tasks to monitor WSL network events and have it run our script everytime it's assigned a new IP address.</p> <p>Watching new entries Event Viewer under <strong>Windows Logs &gt; System</strong> for <strong>Hyper-V</strong>, we can see a couple of new entries when we restart WSL.</p> <p><img src="/posts/fixing-wsl2-localhost-access-issue/wsl-event.png" alt="" /></p> <p>The last entry about loading networking driver seems like a good trigger for our script.</p> <p>Right click the log and click <strong>Attach Task To This Event</strong> and follow the wizard. Choose <strong>Start a program</strong> as an action and type</p> <ul> <li>Program: <code>powershell.exe</code></li> <li>Add arguments: <code>-file c:/path/to/wsl.ps1</code></li> </ul> <p>and complete the wizard.</p> <p>Open Scheduled Tasks and go to <strong>Task Scheduler Library &gt; Event Viewer Tasks</strong>. Find the task we've just created and open its properties.</p> <ul> <li>In <strong>General</strong> tab, check <strong>Run with highest privileges</strong>. Writing inside <code>%WINDIR%</code> requires admin privileges, or the script would fail to run.</li> <li>In <strong>Conditions</strong> tab, disable <strong>Start the task only if the computer is on AC power</strong>.</li> </ul> <p>then save the task.</p> <p>Run the task and check the hosts file to see if WSL IP address is added. Also restart WSL using:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-c"># shutdown</span> <span class="z-n">wsl</span> <span class="z-p">-</span><span class="z-n">-shutdown</span> <span class="z-c"># start</span> <span class="z-n">wsl</span></code></pre> <p>A PowerShell window should pop up and update the IP address.</p> <p>Now we can access servers running on WSL by <code>wsl:PORT</code> when <code>localhost:PORT</code> stops working.</p> Thu, 02 Jul 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/quick-full-text-search-using-sqlite/ https://abdus.dev/posts/quick-full-text-search-using-sqlite/ <p>SQLite provides full-text search functionality with <a href="https://www.sqlite.org/fts5.html" target="_blank" rel="noopener">FTS5 extension</a> that lets us index columns of a table then perform searches using keywords against the table.</p> <h2 id="creating-fts-tables">Creating FTS tables</h2> <p>We'll start with a common scenario, where we keep a set of blog posts in a <code>posts</code> table:</p> <pre data-lang="sql" data-highlighted="true" class="language-sql z-code z-chroma"><code class="language-sql" data-lang="sql"><span class="z-k">CREATE</span><span class="z-w"> </span><span class="z-k">TABLE</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">id</span><span class="z-w"> </span><span class="z-nb">INTEGER</span><span class="z-w"> </span><span class="z-k">NOT</span><span class="z-w"> </span><span class="z-k">NULL</span><span class="z-w"> </span><span class="z-k">PRIMARY</span><span class="z-w"> </span><span class="z-k">KEY</span><span class="z-w"> </span><span class="z-n">AUTOINCREMENT</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">title</span><span class="z-w"> </span><span class="z-nb">TEXT</span><span class="z-w"> </span><span class="z-k">NOT</span><span class="z-w"> </span><span class="z-k">NULL</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">body</span><span class="z-w"> </span><span class="z-nb">TEXT</span><span class="z-w"> </span><span class="z-k">NOT</span><span class="z-w"> </span><span class="z-k">NULL</span><span class="z-w"> </span><span class="z-p">);</span><span class="z-w"> </span></code></pre> <p>We want to conduct searches on <code>title</code> and <code>body</code> columns of a post. Then the corresponding FTS table definition would be:</p> <pre data-lang="sql" data-highlighted="true" class="language-sql z-code z-chroma"><code class="language-sql" data-lang="sql"><span class="z-k">CREATE</span><span class="z-w"> </span><span class="z-n">VIRTUAL</span><span class="z-w"> </span><span class="z-k">TABLE</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-k">USING</span><span class="z-w"> </span><span class="z-n">fts5</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-n">content</span><span class="z-o">=</span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-p">);</span><span class="z-w"> </span></code></pre> <p>Here <code>content=posts</code> indicate that SQLite will query <code>posts</code> table whenever it needs to access column values. By default SQLite uses <code>rowid</code> column to find the row it needs, but an optional <code>content_rowid=id</code> parameter can be specified to instruct to use the preferred key column <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p> <h2 id="keeping-fts-index-up-to-date">Keeping FTS index up-to-date</h2> <p>We've set up FTS indices, but we need to keep it current, and for that we'll create a couple of triggers that hook into <code>posts</code> table and add/update/remove content from the index:</p> <pre data-lang="sql" data-highlighted="true" class="language-sql z-code z-chroma"><code class="language-sql" data-lang="sql"><span class="z-k">CREATE</span><span class="z-w"> </span><span class="z-k">TRIGGER</span><span class="z-w"> </span><span class="z-n">posts_fts_insert</span><span class="z-w"> </span><span class="z-k">AFTER</span><span class="z-w"> </span><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">ON</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-k">BEGIN</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">INTO</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-k">VALUES</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">body</span><span class="z-p">);</span><span class="z-w"> </span><span class="z-k">END</span><span class="z-p">;</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">CREATE</span><span class="z-w"> </span><span class="z-k">TRIGGER</span><span class="z-w"> </span><span class="z-n">posts_fts_delete</span><span class="z-w"> </span><span class="z-k">AFTER</span><span class="z-w"> </span><span class="z-k">DELETE</span><span class="z-w"> </span><span class="z-k">ON</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-k">BEGIN</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">INTO</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-n">posts_fts</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-k">VALUES</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-s1">&#39;delete&#39;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">body</span><span class="z-p">);</span><span class="z-w"> </span><span class="z-k">END</span><span class="z-p">;</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">CREATE</span><span class="z-w"> </span><span class="z-k">TRIGGER</span><span class="z-w"> </span><span class="z-n">posts_fts_update</span><span class="z-w"> </span><span class="z-k">AFTER</span><span class="z-w"> </span><span class="z-k">UPDATE</span><span class="z-w"> </span><span class="z-k">ON</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-k">BEGIN</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">INTO</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-n">posts_fts</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-k">VALUES</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-s1">&#39;delete&#39;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">old</span><span class="z-p">.</span><span class="z-n">body</span><span class="z-p">);</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">INTO</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-k">VALUES</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">rowid</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-k">new</span><span class="z-p">.</span><span class="z-n">body</span><span class="z-p">);</span><span class="z-w"> </span><span class="z-k">END</span><span class="z-p">;</span><span class="z-w"> </span></code></pre> <p>Now we can insert new records into <code>posts</code> table and FTS index will be populated automatically.</p> <pre data-lang="sql" data-highlighted="true" class="language-sql z-code z-chroma"><code class="language-sql" data-lang="sql"><span class="z-k">INSERT</span><span class="z-w"> </span><span class="z-k">INTO</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-p">(</span><span class="z-n">title</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-n">body</span><span class="z-p">)</span><span class="z-w"> </span><span class="z-k">VALUES</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-s1">&#39;I daresay that Fry has discovered the smelliest object in the known universe!&#39;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-s1">&#39;Soothe us with sweet lies. You&#39;&#39;ve killed me! Oh, you&#39;&#39;ve killed me! Good news, everyone! There&#39;&#39;s a report on TV with some very bad news! Look, last night was a mistake.&#39;</span><span class="z-p">),</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-p">(</span><span class="z-s1">&#39;Have you ever tried just turning off the TV, sitting down with your children, and hitting them?&#39;</span><span class="z-p">,</span><span class="z-w"> </span><span class="z-w"> </span><span class="z-s1">&#39;Yes. You gave me a dollar and some candy. Professor, make a woman out of me. You seem malnourished. Are you suffering from intestinal parasites? Is the Space Pope reptilian!? I don&#39;&#39;t &#39;&#39;need&#39;&#39; to drink. I can quit anytime I want!&#39;</span><span class="z-p">);</span><span class="z-w"> </span></code></pre> <h2 id="querying-fts-table">Querying FTS table</h2> <p>Now we can perform full-text searches using FTS table to find matching records, then get the content for those.</p> <pre data-lang="sql" data-highlighted="true" class="language-sql z-code z-chroma"><code class="language-sql" data-lang="sql"><span class="z-k">SELECT</span><span class="z-w"> </span><span class="z-o">*</span><span class="z-w"> </span><span class="z-k">FROM</span><span class="z-w"> </span><span class="z-n">posts</span><span class="z-w"> </span><span class="z-k">WHERE</span><span class="z-w"> </span><span class="z-n">ROWID</span><span class="z-w"> </span><span class="z-k">IN</span><span class="z-w"> </span><span class="z-p">(</span><span class="z-k">SELECT</span><span class="z-w"> </span><span class="z-n">ROWID</span><span class="z-w"> </span><span class="z-k">FROM</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-k">WHERE</span><span class="z-w"> </span><span class="z-n">posts_fts</span><span class="z-w"> </span><span class="z-k">MATCH</span><span class="z-w"> </span><span class="z-s1">&#39;fry&#39;</span><span class="z-w"> </span><span class="z-k">ORDER</span><span class="z-w"> </span><span class="z-k">BY</span><span class="z-w"> </span><span class="z-n">rank</span><span class="z-p">);</span><span class="z-w"> </span></code></pre> <p>Here we use <code>&lt;fts_table&gt; match 'term'</code> to search FTS table and order the results according to their relevance using <code>rank</code> value.</p> <p>this gives us a match as expected:</p> <pre data-lang="commandline" data-highlighted="true" class="language-commandline z-code z-chroma"><code class="language-commandline" data-lang="commandline">sqlite&gt; .mode line sqlite&gt; SELECT * ...&gt; FROM posts ...&gt; WHERE ROWID IN (SELECT ROWID FROM posts_fts WHERE posts_fts MATCH &#39;fry&#39; ORDER BY rank); id = 1 title = I daresay that Fry has discovered the smelliest object in the known universe! body = Soothe us with sweet lies. You&#39;ve killed me! Oh, you&#39;ve killed me! Good news, everyone! There&#39;s a report on TV with some very bad news! Look, last night was a mistake.</code></pre> <p>SQLite supports a couple of operators in addition to plain search, such as <code>AND</code>, <code>OR</code>, <code>NOT</code>, verbatim <code>&quot;term&quot;</code> <sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p> <h2 id="python-script">Python script</h2> <p>Here's a python script to enable FTS for a table and search against it. Loosely inspired by <a href="https://github.com/simonw/sqlite-utils" target="_blank" rel="noopener">sqlite-utils</a></p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">sqlite3</span> <span class="z-kn">from</span> <span class="z-nn">sqlite3</span> <span class="z-kn">import</span> <span class="z-n">Connection</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">List</span> <span class="z-k">def</span> <span class="z-nf">enable_fts</span><span class="z-p">(</span><span class="z-n">db</span><span class="z-p">:</span> <span class="z-n">Connection</span><span class="z-p">,</span> <span class="z-n">table</span><span class="z-p">:</span> <span class="z-nb">str</span><span class="z-p">,</span> <span class="z-n">columns</span><span class="z-p">:</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-nb">str</span><span class="z-p">]):</span> <span class="z-n">column_list</span> <span class="z-o">=</span> <span class="z-s1">&#39;,&#39;</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;[</span><span class="z-si">{</span><span class="z-n">c</span><span class="z-si">}</span><span class="z-s1">]&#39;</span> <span class="z-k">for</span> <span class="z-n">c</span> <span class="z-ow">in</span> <span class="z-n">columns</span><span class="z-p">)</span> <span class="z-n">db</span><span class="z-o">.</span><span class="z-n">executescript</span><span class="z-p">(</span><span class="z-s1">&#39;&#39;&#39; </span><span class="z-s1"> CREATE VIRTUAL TABLE [</span><span class="z-si">{table}</span><span class="z-s1">_fts] USING fts5 </span><span class="z-s1"> ( </span><span class="z-s1"> </span><span class="z-si">{column_list}</span><span class="z-s1">, </span><span class="z-s1"> content=[</span><span class="z-si">{table}</span><span class="z-s1">_fts] </span><span class="z-s1"> )&#39;&#39;&#39;</span><span class="z-o">.</span><span class="z-n">format</span><span class="z-p">(</span> <span class="z-n">table</span><span class="z-o">=</span><span class="z-n">table</span><span class="z-p">,</span> <span class="z-n">column_list</span><span class="z-o">=</span><span class="z-n">column_list</span> <span class="z-p">))</span> <span class="z-n">db</span><span class="z-o">.</span><span class="z-n">executescript</span><span class="z-p">(</span><span class="z-s1">&#39;&#39;&#39; </span><span class="z-s1"> CREATE TRIGGER [</span><span class="z-si">{table}</span><span class="z-s1">_fts_insert] AFTER INSERT ON posts </span><span class="z-s1"> BEGIN </span><span class="z-s1"> INSERT INTO [</span><span class="z-si">{table}</span><span class="z-s1">_fts] (rowid, </span><span class="z-si">{column_list}</span><span class="z-s1">) VALUES (new.rowid, </span><span class="z-si">{new_columns}</span><span class="z-s1">); </span><span class="z-s1"> END; </span><span class="z-s1"> CREATE TRIGGER [</span><span class="z-si">{table}</span><span class="z-s1">_fts_delete] AFTER DELETE ON posts </span><span class="z-s1"> BEGIN </span><span class="z-s1"> INSERT INTO [</span><span class="z-si">{table}</span><span class="z-s1">_fts] ([</span><span class="z-si">{table}</span><span class="z-s1">_fts], rowid, </span><span class="z-si">{column_list}</span><span class="z-s1">) VALUES (&#39;delete&#39;, old.rowid, </span><span class="z-si">{old_columns}</span><span class="z-s1">); </span><span class="z-s1"> END; </span><span class="z-s1"> CREATE TRIGGER [</span><span class="z-si">{table}</span><span class="z-s1">_fts_update] AFTER UPDATE ON posts </span><span class="z-s1"> BEGIN </span><span class="z-s1"> INSERT INTO [</span><span class="z-si">{table}</span><span class="z-s1">_fts] ([</span><span class="z-si">{table}</span><span class="z-s1">_fts], rowid, </span><span class="z-si">{column_list}</span><span class="z-s1">) VALUES (&#39;delete&#39;, old.rowid, </span><span class="z-si">{old_columns}</span><span class="z-s1">); </span><span class="z-s1"> INSERT INTO [</span><span class="z-si">{table}</span><span class="z-s1">_fts] (rowid, </span><span class="z-si">{column_list}</span><span class="z-s1">) VALUES (new.rowid, </span><span class="z-si">{new_columns}</span><span class="z-s1">); </span><span class="z-s1"> END; </span><span class="z-s1"> &#39;&#39;&#39;</span><span class="z-o">.</span><span class="z-n">format</span><span class="z-p">(</span> <span class="z-n">table</span><span class="z-o">=</span><span class="z-n">table</span><span class="z-p">,</span> <span class="z-n">column_list</span><span class="z-o">=</span><span class="z-n">column_list</span><span class="z-p">,</span> <span class="z-n">new_columns</span><span class="z-o">=</span><span class="z-s1">&#39;,&#39;</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;new.[</span><span class="z-si">{</span><span class="z-n">c</span><span class="z-si">}</span><span class="z-s1">]&#39;</span> <span class="z-k">for</span> <span class="z-n">c</span> <span class="z-ow">in</span> <span class="z-n">columns</span><span class="z-p">),</span> <span class="z-n">old_columns</span><span class="z-o">=</span><span class="z-s1">&#39;,&#39;</span><span class="z-o">.</span><span class="z-n">join</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;old.[</span><span class="z-si">{</span><span class="z-n">c</span><span class="z-si">}</span><span class="z-s1">]&#39;</span> <span class="z-k">for</span> <span class="z-n">c</span> <span class="z-ow">in</span> <span class="z-n">columns</span><span class="z-p">),</span> <span class="z-p">))</span> <span class="z-k">def</span> <span class="z-nf">query</span><span class="z-p">(</span><span class="z-n">db</span><span class="z-p">:</span> <span class="z-n">Connection</span><span class="z-p">,</span> <span class="z-n">table</span><span class="z-p">:</span> <span class="z-nb">str</span><span class="z-p">,</span> <span class="z-n">term</span><span class="z-p">:</span> <span class="z-nb">str</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">sqlite3</span><span class="z-o">.</span><span class="z-n">Row</span><span class="z-p">]:</span> <span class="z-n">cur</span> <span class="z-o">=</span> <span class="z-n">db</span><span class="z-o">.</span><span class="z-n">execute</span><span class="z-p">(</span><span class="z-s1">&#39;&#39;&#39; </span><span class="z-s1"> SELECT * FROM [</span><span class="z-si">{table}</span><span class="z-s1">] </span><span class="z-s1"> WHERE ROWID IN (SELECT ROWID FROM [</span><span class="z-si">{table}</span><span class="z-s1">_fts] WHERE [</span><span class="z-si">{table}</span><span class="z-s1">_fts] MATCH ? ORDER BY rank) </span><span class="z-s1"> &#39;&#39;&#39;</span><span class="z-o">.</span><span class="z-n">format</span><span class="z-p">(</span><span class="z-n">table</span><span class="z-o">=</span><span class="z-n">table</span><span class="z-p">),</span> <span class="z-p">[</span><span class="z-n">term</span><span class="z-p">])</span> <span class="z-k">return</span> <span class="z-nb">list</span><span class="z-p">(</span><span class="z-n">cur</span><span class="z-o">.</span><span class="z-n">fetchall</span><span class="z-p">())</span></code></pre> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>More on <a href="https://www.sqlite.org/fts5.html#external_content_tables" target="_blank" rel="noopener">external content</a>&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>More on <a href="https://www.sqlite.org/fts5.html#full_text_query_syntax" target="_blank" rel="noopener">search syntax</a>&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Wed, 01 Jul 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/building-a-ui-with-vanilla-javascript/ https://abdus.dev/posts/building-a-ui-with-vanilla-javascript/ <p>I enjoy building server-rendered apps. They're faster to build, more straightforward, and easy to get your head around.</p> <p>One tool I keep reaching for is <a href="https://github.com/alpinejs/alpine/" target="_blank" rel="noopener">Alpine JS</a>, which is a tiny library that helps to build components on a server-rendered page with a Vue-like syntax.</p> <p>But it does not apply to all needs, or sometimes a couple of lines of vanilla JS is all the page ever needs. In those times, I write this utility function:</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-cm">/** </span><span class="z-cm"> * @param {string} name </span><span class="z-cm"> * @param {HTMLElement.prototype} props </span><span class="z-cm"> * @param {Array&lt;HTMLElement|string&gt;} children </span><span class="z-cm"> * @return HTMLElement </span><span class="z-cm"> */</span> <span class="z-kd">function</span> <span class="z-nx">$h</span><span class="z-p">(</span><span class="z-nx">name</span> <span class="z-o">=</span> <span class="z-s1">&#39;div&#39;</span><span class="z-p">,</span> <span class="z-nx">props</span> <span class="z-o">=</span> <span class="z-p">{},</span> <span class="z-nx">children</span> <span class="z-o">=</span> <span class="z-p">[])</span> <span class="z-p">{</span> <span class="z-kr">const</span> <span class="z-nx">el</span> <span class="z-o">=</span> <span class="z-nb">document</span><span class="z-p">.</span><span class="z-nx">createElement</span><span class="z-p">(</span><span class="z-nx">name</span><span class="z-p">);</span> <span class="z-nb">Object</span><span class="z-p">.</span><span class="z-nx">assign</span><span class="z-p">(</span><span class="z-nx">el</span><span class="z-p">,</span> <span class="z-nx">props</span><span class="z-p">);</span> <span class="z-nx">el</span><span class="z-p">.</span><span class="z-nx">append</span><span class="z-p">(...</span><span class="z-nx">children</span><span class="z-p">);</span> <span class="z-k">return</span> <span class="z-nx">el</span><span class="z-p">;</span> <span class="z-p">}</span></code></pre> <p>and use it like so:</p> <pre data-lang="js" data-highlighted="true" class="language-js z-code z-chroma"><code class="language-js" data-lang="js"><span class="z-kr">const</span> <span class="z-nx">button</span> <span class="z-o">=</span> <span class="z-nx">$h</span><span class="z-p">(</span><span class="z-s1">&#39;button&#39;</span><span class="z-p">,</span> <span class="z-p">{</span> <span class="z-nx">className</span><span class="z-o">:</span> <span class="z-s1">&#39;button&#39;</span><span class="z-p">,</span> <span class="z-nx">onclick</span><span class="z-o">:</span> <span class="z-kr">async</span> <span class="z-p">(</span><span class="z-nx">e</span><span class="z-p">)</span> <span class="z-p">=&gt;</span> <span class="z-nx">alert</span><span class="z-p">(</span><span class="z-s1">&#39;clicked&#39;</span><span class="z-p">),</span> <span class="z-p">},</span> <span class="z-p">[</span><span class="z-s1">&#39;Download&#39;</span><span class="z-p">]);</span> <span class="z-nb">document</span><span class="z-p">.</span><span class="z-nx">body</span><span class="z-p">.</span><span class="z-nx">append</span><span class="z-p">(</span><span class="z-nx">button</span><span class="z-p">);</span></code></pre> <p>It resembles <a href="https://mithril.js.org/index.html#dom-elements" target="_blank" rel="noopener">Mithril's syntax</a>, but it's not reactive as modern UI libraries.</p> Mon, 29 Jun 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/ffmpeg-metadata/ https://abdus.dev/posts/ffmpeg-metadata/ <p>I've been maintaining a movie archive for years now. Every year or so I go over what I have and do a clean-up if necessary.</p> <p>This time I've decided to add metadata about the title, year, IMDb links, artists, etc. to videos. I've written about <a href="/posts/powershell-file-metadata-guide/#video-metadata">a way to do it on PowerShell</a>, but nothing comes closer to the joy I get from writing in Python.</p> <h2 id="ffmpeg-command">ffmpeg command</h2> <p>ffmpeg supports practically anything under the sun, including adding metadata. <a href="https://stackoverflow.com/a/11479066/5298150" target="_blank" rel="noopener">This answer</a> on StackOverflow put me on the right path. With ffmpeg, adding metadata is simply including <code>-metadata key=value</code> arguments in the command:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">ffmpeg</span> <span class="z-n">-i</span> <span class="z-n">video</span><span class="z-p">.</span><span class="z-py">mkv</span> <span class="z-n">-c</span> <span class="z-nb">copy </span><span class="z-n">-movflags</span> <span class="z-n">use_metadata_tags</span> <span class="z-n">-map_metadata</span> <span class="z-mf">0</span> <span class="z-n">-metadata</span> <span class="z-n">title</span><span class="z-p">=</span><span class="z-s2">&#34;Hello World&#34;</span> <span class="z-n">-metadata</span> <span class="z-n">year</span><span class="z-p">=</span><span class="z-mf">2020</span> <span class="z-n">video</span><span class="z-p">.</span><span class="z-py">metadata</span><span class="z-p">.</span><span class="z-py">mkv</span></code></pre> <p>when split into lines:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">ffmpeg</span> <span class="z-n">-i</span> <span class="z-n">video</span><span class="z-p">.</span><span class="z-py">mkv</span> <span class="z-p">`</span> <span class="z-n">-c</span> <span class="z-nb">copy </span><span class="z-p">`</span> <span class="z-n">-movflags</span> <span class="z-n">use_metadata_tags</span> <span class="z-p">`</span> <span class="z-n">-map_metadata</span> <span class="z-mf">0</span> <span class="z-p">`</span> <span class="z-n">-metadata</span> <span class="z-n">title</span><span class="z-p">=</span><span class="z-s2">&#34;Hello World&#34;</span> <span class="z-p">`</span> <span class="z-n">-metadata</span> <span class="z-n">year</span><span class="z-p">=</span><span class="z-mf">2020</span> <span class="z-p">`</span> <span class="z-n">video</span><span class="z-p">.</span><span class="z-py">metadata</span><span class="z-p">.</span><span class="z-py">mkv</span></code></pre> <p>Some important points:</p> <ul> <li>Use <code>-c copy</code> to copy streams directly without encoding.</li> <li>Use <code>-map_metadata 0</code> to copy over existing meta tags without changing them<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</li> </ul> <p>Not all containers support every metadata. For reference you can check <a href="https://www.matroska.org/technical/tagging.html" target="_blank" rel="noopener">Matroska spec</a> for MKV and <a href="https://kodi.wiki/view/Video_file_tagging#Supported_Tags" target="_blank" rel="noopener">Kodi docs</a> for MP4.</p> <h2 id="python-script">Python script</h2> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">from</span> <span class="z-nn">pathlib</span> <span class="z-kn">import</span> <span class="z-n">Path</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Dict</span> <span class="z-k">def</span> <span class="z-nf">add_metadata</span><span class="z-p">(</span><span class="z-n">video</span><span class="z-p">:</span> <span class="z-n">Path</span><span class="z-p">,</span> <span class="z-n">meta</span><span class="z-p">:</span> <span class="z-n">Dict</span><span class="z-p">[</span><span class="z-nb">str</span><span class="z-p">,</span> <span class="z-nb">str</span><span class="z-p">],</span> <span class="z-n">save_path</span><span class="z-p">:</span> <span class="z-n">Path</span> <span class="z-o">=</span> <span class="z-kc">None</span><span class="z-p">,</span> <span class="z-n">overwrite</span><span class="z-p">:</span> <span class="z-nb">bool</span> <span class="z-o">=</span> <span class="z-kc">True</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-ow">not</span> <span class="z-n">save_path</span><span class="z-p">:</span> <span class="z-n">save_path</span> <span class="z-o">=</span> <span class="z-n">video</span><span class="z-o">.</span><span class="z-n">with_suffix</span><span class="z-p">(</span><span class="z-s1">&#39;.metadata&#39;</span> <span class="z-o">+</span> <span class="z-n">video</span><span class="z-o">.</span><span class="z-n">suffix</span><span class="z-p">)</span> <span class="z-n">metadata_args</span> <span class="z-o">=</span> <span class="z-p">[]</span> <span class="z-k">for</span> <span class="z-n">k</span><span class="z-p">,</span> <span class="z-n">v</span> <span class="z-ow">in</span> <span class="z-n">meta</span><span class="z-o">.</span><span class="z-n">items</span><span class="z-p">():</span> <span class="z-n">metadata_args</span><span class="z-o">.</span><span class="z-n">extend</span><span class="z-p">([</span> <span class="z-s1">&#39;-metadata&#39;</span><span class="z-p">,</span> <span class="z-sa">f</span><span class="z-s1">&#39;</span><span class="z-si">{</span><span class="z-n">k</span><span class="z-si">}</span><span class="z-s1">=</span><span class="z-si">{</span><span class="z-n">v</span><span class="z-si">}</span><span class="z-s1">&#39;</span> <span class="z-p">])</span> <span class="z-n">args</span> <span class="z-o">=</span> <span class="z-p">[</span> <span class="z-s1">&#39;ffmpeg&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-v&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;quiet&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-i&#39;</span><span class="z-p">,</span> <span class="z-nb">str</span><span class="z-p">(</span><span class="z-n">video</span><span class="z-o">.</span><span class="z-n">absolute</span><span class="z-p">()),</span> <span class="z-s1">&#39;-movflags&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;use_metadata_tags&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-map_metadata&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;0&#39;</span><span class="z-p">,</span> <span class="z-o">*</span><span class="z-n">metadata_args</span><span class="z-p">,</span> <span class="z-s1">&#39;-c&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;copy&#39;</span><span class="z-p">,</span> <span class="z-nb">str</span><span class="z-p">(</span><span class="z-n">save_path</span><span class="z-p">)</span> <span class="z-p">]</span> <span class="z-k">if</span> <span class="z-n">overwrite</span><span class="z-p">:</span> <span class="z-n">args</span><span class="z-o">.</span><span class="z-n">append</span><span class="z-p">(</span><span class="z-s1">&#39;-y&#39;</span><span class="z-p">)</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">run</span><span class="z-p">(</span><span class="z-n">args</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span><span class="z-p">)</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">check_returncode</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">vid</span> <span class="z-o">=</span> <span class="z-n">Path</span><span class="z-p">(</span><span class="z-sa">r</span><span class="z-s1">&#39;video.mp4&#39;</span><span class="z-p">)</span> <span class="z-n">add_metadata</span><span class="z-p">(</span> <span class="z-n">vid</span><span class="z-p">,</span> <span class="z-n">meta</span><span class="z-o">=</span><span class="z-nb">dict</span><span class="z-p">(</span> <span class="z-n">title</span><span class="z-o">=</span><span class="z-n">vid</span><span class="z-o">.</span><span class="z-n">stem</span><span class="z-p">,</span> <span class="z-n">comment</span><span class="z-o">=</span><span class="z-n">vid</span><span class="z-o">.</span><span class="z-n">stem</span><span class="z-p">,</span> <span class="z-n">year</span><span class="z-o">=</span><span class="z-mi">2020</span><span class="z-p">,</span> <span class="z-p">),</span> <span class="z-p">)</span></code></pre> <p><code>add_metadata</code> function takes arbitrary key:value parameters and turns them into ffmpeg arguments.</p> <div class="footnotes" role="doc-endnotes"> <hr /> <ol> <li id="fn:1"> <p>From <a href="https://video.stackexchange.com/questions/23741/how-to-prevent-ffmpeg-from-dropping-metadata" target="_blank" rel="noopener">this</a> StackOverflow answer&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div> Sun, 28 Jun 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/python-monitor-usb/ https://abdus.dev/posts/python-monitor-usb/ <p>I was looking for a way to monitor USB drives and trigger a backup when I plug in my backup drive. Most solutions online describe a way to do it in C# or C++, but I wanted to write one in Python.</p> <p><a href="#alternative-approach-polling-for-drives">My first naive attempt</a> was to get the list of drives, and poll for changes every couple of seconds. I've explained this method below and it works fine.</p> <p>However I wasn't really comfortable with launching a subprocess every second. It seemed wasteful. So I've written a second method that uses Win32 APIs to hook into Windows and monitor drive changes.</p> <div class="has-recommend"> <p>If you're interested, I have <a href="/posts/monitor-clipboard/">another post</a> where I build a clipboard listener that you can use to monitor clipboard contents.</p> <div class="recommend"> <p><a href="/posts/monitor-clipboard/">Monitoring clipboard contents on Windows with Python</a></p> </div> </div> <h2 id="getting-a-list-of-drives">Getting a list of drives</h2> <p>Windows provides an interface for sysadmins called <a href="https://docs.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page" target="_blank" rel="noopener">Windows Management Instrumentation</a> (WMI) which basically allows you to monitor a local or remote system.</p> <p>On PowerShell we can use <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-wmiobject" target="_blank" rel="noopener"><code>Get-WmiObject</code></a> command to interact with it. WMI <a href="https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/win32-logicaldisk" target="_blank" rel="noopener"><code>Win32_LogicalDisk</code></a> class gives us a list of the drives currently connected to the PC.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nb">PS </span><span class="z-p">&gt;</span> <span class="z-nb">Get-WmiObject</span> <span class="z-n">-Class</span> <span class="z-n">Win32_LogicalDisk</span> <span class="z-n">DeviceID</span> <span class="z-err">:</span> <span class="z-n">C:</span> <span class="z-n">DriveType</span> <span class="z-err">:</span> <span class="z-mf">3</span> <span class="z-n">ProviderName</span> <span class="z-err">:</span> <span class="z-n">FreeSpace</span> <span class="z-err">:</span> <span class="z-mf">21341257728</span> <span class="z-n">Size</span> <span class="z-err">:</span> <span class="z-mf">130417745920</span> <span class="z-n">VolumeName</span> <span class="z-err">:</span> <span class="z-p">...</span> <span class="z-n">DeviceID</span> <span class="z-err">:</span> <span class="z-n">E:</span> <span class="z-n">DriveType</span> <span class="z-err">:</span> <span class="z-mf">2</span> <span class="z-n">ProviderName</span> <span class="z-err">:</span> <span class="z-n">FreeSpace</span> <span class="z-err">:</span> <span class="z-mf">48837689344</span> <span class="z-n">Size</span> <span class="z-err">:</span> <span class="z-mf">62706155520</span> <span class="z-n">VolumeName</span> <span class="z-err">:</span> <span class="z-n">ABDUS</span></code></pre> <p>That's a lot of useful information 👌. We can use <a href="https://devblogs.microsoft.com/scripting/inventory-drive-types-by-using-powershell/" target="_blank" rel="noopener"><code>DriveType</code></a> to determine if a drive is removable or not.</p> <div class="table"> <table> <thead> <tr> <th align="right">DriveType</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td align="right"><code>0</code></td> <td>Unknown</td> </tr> <tr> <td align="right"><code>1</code></td> <td>No Root Directory</td> </tr> <tr> <td align="right"><strong><code>2</code></strong></td> <td><strong>Removable Disk</strong></td> </tr> <tr> <td align="right"><code>3</code></td> <td>Local Disk</td> </tr> <tr> <td align="right"><code>4</code></td> <td>Network Drive</td> </tr> <tr> <td align="right"><code>5</code></td> <td>Compact Disc</td> </tr> <tr> <td align="right"><code>6</code></td> <td>RAM Disk</td> </tr> </tbody> </table> </div> <p>Finally, we can pick the fields we need and convert the command result to JSON so that Python can easily parse it.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nb">PS </span><span class="z-p">&gt;</span> <span class="z-nb">Get-WmiObject</span> <span class="z-n">-Class</span> <span class="z-n">Win32_LogicalDisk</span> <span class="z-p">|</span> <span class="z-nb">Select-Object</span> <span class="z-n">deviceid</span><span class="z-p">,</span><span class="z-n">volumename</span><span class="z-p">,</span><span class="z-n">drivetype</span> <span class="z-p">|</span> <span class="z-nb">ConvertTo-Json</span></code></pre> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;DeviceID&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;C:&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;VolumeName&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;DriveType&#34;</span><span class="z-p">:</span> <span class="z-mi">3</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;DeviceID&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;E:&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;VolumeName&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;ABDUS&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;DriveType&#34;</span><span class="z-p">:</span> <span class="z-mi">2</span> <span class="z-p">},</span> <span class="z-err">...</span> <span class="z-p">]</span></code></pre> <p>Now we can call this PowerShell command from Python using <a href="https://docs.python.org/3/library/subprocess.html#subprocess.run" target="_blank" rel="noopener"><code>subprocess</code></a> module and parse its JSON output:</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">from</span> <span class="z-nn">dataclasses</span> <span class="z-kn">import</span> <span class="z-n">dataclass</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span><span class="z-p">,</span> <span class="z-n">List</span> <span class="z-nd">@dataclass</span> <span class="z-k">class</span> <span class="z-nc">Drive</span><span class="z-p">:</span> <span class="z-n">letter</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">label</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">drive_type</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-nd">@property</span> <span class="z-k">def</span> <span class="z-nf">is_removable</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-nb">bool</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">drive_type</span> <span class="z-o">==</span> <span class="z-s1">&#39;Removable Disk&#39;</span> <span class="z-k">def</span> <span class="z-nf">list_drives</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Get a list of drives using WMI </span><span class="z-s2"> :return: list of drives </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">run</span><span class="z-p">(</span> <span class="z-n">args</span><span class="z-o">=</span><span class="z-p">[</span> <span class="z-s1">&#39;powershell&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-noprofile&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-command&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;Get-WmiObject -Class Win32_LogicalDisk | Select-Object deviceid,volumename,drivetype | ConvertTo-Json&#39;</span> <span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span> <span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">returncode</span> <span class="z-o">!=</span> <span class="z-mi">0</span> <span class="z-ow">or</span> <span class="z-ow">not</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">():</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-s1">&#39;Failed to enumerate drives&#39;</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-p">[]</span> <span class="z-n">devices</span> <span class="z-o">=</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-p">)</span> <span class="z-n">drive_types</span> <span class="z-o">=</span> <span class="z-p">{</span> <span class="z-mi">0</span><span class="z-p">:</span> <span class="z-s1">&#39;Unknown&#39;</span><span class="z-p">,</span> <span class="z-mi">1</span><span class="z-p">:</span> <span class="z-s1">&#39;No Root Directory&#39;</span><span class="z-p">,</span> <span class="z-mi">2</span><span class="z-p">:</span> <span class="z-s1">&#39;Removable Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">3</span><span class="z-p">:</span> <span class="z-s1">&#39;Local Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">4</span><span class="z-p">:</span> <span class="z-s1">&#39;Network Drive&#39;</span><span class="z-p">,</span> <span class="z-mi">5</span><span class="z-p">:</span> <span class="z-s1">&#39;Compact Disc&#39;</span><span class="z-p">,</span> <span class="z-mi">6</span><span class="z-p">:</span> <span class="z-s1">&#39;RAM Disk&#39;</span><span class="z-p">,</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">(</span> <span class="z-n">letter</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;deviceid&#39;</span><span class="z-p">],</span> <span class="z-n">label</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;volumename&#39;</span><span class="z-p">],</span> <span class="z-n">drive_type</span><span class="z-o">=</span><span class="z-n">drive_types</span><span class="z-p">[</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;drivetype&#39;</span><span class="z-p">]]</span> <span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">d</span> <span class="z-ow">in</span> <span class="z-n">devices</span><span class="z-p">]</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-n">list_drives</span><span class="z-p">())</span></code></pre> <p>which outputs</p> <pre><code>[Drive(letter='C:', label='', drive_type='Local Disk'), ...] </code></pre> <p>OK. Now the difficult part: hooking it up to Windows.</p> <h2 id="listening-to-windows-wm-devicechange-messages">Listening to Windows <code>WM_DEVICECHANGE</code> messages</h2> <p>Windows uses <a href="https://docs.microsoft.com/en-us/windows/win32/learnwin32/window-messages" target="_blank" rel="noopener">messages</a> to notify programs of events and let them react. This includes user inputs (mouse clicks, key strokes etc.) and other OS events (hardware connected, low power, A/C adapter connected etc.).</p> <p>Windows broadcasts a <a href="https://docs.microsoft.com/en-us/windows/win32/devio/wm-devicechange#parameters" target="_blank" rel="noopener"><code>WM_DEVICECHANGE</code> message</a> when hardware configuration of the system changes. That includes plug-and-play devices, such as USB drives, printers, mouse etc. We need a way to listen to this broadcast. That requires creating a window, registering a window procedure, then running a message loop that receives messages from the operating system.</p> <p>We need the <a href="https://github.com/mhammond/pywin32" target="_blank" rel="noopener"><code>pywin32</code></a> package, which provides extensions to consume Win32 APIs in Python. We can install it with pip:</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell">pip install pywin32</code></pre> <p>To create a window, we can follow <a href="https://docs.microsoft.com/en-us/windows/win32/learnwin32/creating-a-window" target="_blank" rel="noopener">Microsoft's docs</a> and translate the given C++ example to its Python equivalent.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">win32api</span><span class="z-o">,</span> <span class="z-nn">win32con</span><span class="z-o">,</span> <span class="z-nn">win32gui</span> <span class="z-k">def</span> <span class="z-nf">create_window</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-nb">int</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Create a window for listening to messages </span><span class="z-s2"> https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-createwindowexa#example </span><span class="z-s2"> </span><span class="z-s2"> See also: https://docs.microsoft.com/en-us/windows/win32/learnwin32/creating-a-window#creating-the-window </span><span class="z-s2"> </span><span class="z-s2"> :return: window hwnd </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">wc</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">WNDCLASS</span><span class="z-p">()</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpfnWndProc</span> <span class="z-o">=</span> <span class="z-nb">print</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpszClassName</span> <span class="z-o">=</span> <span class="z-s1">&#39;Demo&#39;</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span> <span class="z-o">=</span> <span class="z-n">win32api</span><span class="z-o">.</span><span class="z-n">GetModuleHandle</span><span class="z-p">(</span><span class="z-kc">None</span><span class="z-p">)</span> <span class="z-n">class_atom</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">RegisterClass</span><span class="z-p">(</span><span class="z-n">wc</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">CreateWindow</span><span class="z-p">(</span><span class="z-n">class_atom</span><span class="z-p">,</span> <span class="z-s1">&#39;Demo&#39;</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span><span class="z-p">,</span> <span class="z-kc">None</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">create_window</span><span class="z-p">()</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span></code></pre> <p>The highlighted line is the critical part. It is our <a href="https://docs.microsoft.com/en-us/windows/win32/winmsg/using-window-procedures" target="_blank" rel="noopener">&quot;window procedure&quot;</a> that gets called when the window receives a message. It must have the following signature:</p> <pre data-lang="c++" data-highlighted="true" class="language-c++ z-code z-chroma"><code class="language-c++" data-lang="c++"><span class="z-n">LRESULT</span> <span class="z-n">CALLBACK</span> <span class="z-nf">MainWndProc</span><span class="z-p">(</span> <span class="z-n">HWND</span> <span class="z-n">hwnd</span><span class="z-p">,</span> <span class="z-c1">// handle to window </span> <span class="z-n">UINT</span> <span class="z-n">uMsg</span><span class="z-p">,</span> <span class="z-c1">// message identifier </span> <span class="z-n">WPARAM</span> <span class="z-n">wParam</span><span class="z-p">,</span> <span class="z-c1">// first message parameter </span> <span class="z-n">LPARAM</span> <span class="z-n">lParam</span><span class="z-p">)</span> <span class="z-c1">// second message parameter </span><span class="z-p">{</span> <span class="z-p">...</span> <span class="z-p">}</span></code></pre> <p>Python's <code>print</code> function accepts an arbitrary number of parameters and dumps all its arguments to console, so its signature is compatible.</p> <p>Once we run this script we can see the messages received by our window:</p> <pre><code>789538 537 7 0 789538 537 32772 346174713248 ... </code></pre> <p>All these integers are either a value or a pointer to a value. Let's break down the second message:</p> <div class="table"> <table> <thead> <tr> <th>Parameter</th> <th>Value</th> <th>Hexadecimal</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>hwnd</code></td> <td><code>789538</code></td> <td><code>0xc0c22</code></td> <td>our window's handle</td> </tr> <tr> <td><code>msg</code></td> <td><code>537</code></td> <td><code>0x0219</code></td> <td><a href="https://docs.microsoft.com/en-us/windows/win32/devio/wm-devicechange#parameters" target="_blank" rel="noopener"><code>WM_DEVICECHANGE</code></a> message</td> </tr> <tr> <td><code>wparam</code></td> <td><code>32772</code></td> <td><code>0x8004</code></td> <td><a href="https://docs.microsoft.com/en-us/windows/win32/devio/dbt-deviceremovecomplete" target="_blank" rel="noopener"><code>DBT_DEVICEREMOVECOMPLETE</code></a> event</td> </tr> <tr> <td><code>lparam</code></td> <td><code>346174713248</code></td> <td><code>0x50999eeda0</code></td> <td>pointer (memory address) to event info</td> </tr> </tbody> </table> </div> <p>Once we decipher the message type, we can google its hexadecimal value to find which message it corresponds to, then figure out what it contains. Here we don't really need to dereference and unpack <code>lparam</code> pointer, because <a href="#getting-a-list-of-drives">we're using WMI</a> for that. We just need to know something has happened.</p> <p><code>WM_DEVICECHANGE</code> message gives us <code>DBT_DEVICEARRIVAL</code> and <code>DBT_DEVICEREMOVECOMPLETE</code> events to notify when a device is added or removed respectively.</p> <p>Now putting these all together:</p> <h2 id="python-script">Python script</h2> <p>I wrapped the code I've explained above inside <code>DeviceListener</code> class. It provides an easy way to attach your own listener and perform a task when a drive is attached/removed.</p> <p>It assumes the script will be run as the main script, that's why <code>DeviceListener.start()</code> is blocking. You can run it inside a thread if you want it to be non-blocking.</p> <p>You can also change the highlighted line to trigger the callback only when a drive is either inserted or removed.</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">import</span> <span class="z-nn">logging</span> <span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">from</span> <span class="z-nn">dataclasses</span> <span class="z-kn">import</span> <span class="z-n">dataclass</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span><span class="z-p">,</span> <span class="z-n">List</span> <span class="z-kn">import</span> <span class="z-nn">win32api</span><span class="z-o">,</span> <span class="z-nn">win32con</span><span class="z-o">,</span> <span class="z-nn">win32gui</span> <span class="z-n">logging</span><span class="z-o">.</span><span class="z-n">basicConfig</span><span class="z-p">(</span><span class="z-n">level</span><span class="z-o">=</span><span class="z-n">logging</span><span class="z-o">.</span><span class="z-n">INFO</span><span class="z-p">)</span> <span class="z-n">logger</span> <span class="z-o">=</span> <span class="z-n">logging</span><span class="z-o">.</span><span class="z-n">getLogger</span><span class="z-p">(</span><span class="z-vm">__name__</span><span class="z-p">)</span> <span class="z-nd">@dataclass</span> <span class="z-k">class</span> <span class="z-nc">Drive</span><span class="z-p">:</span> <span class="z-n">letter</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">label</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">drive_type</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-nd">@property</span> <span class="z-k">def</span> <span class="z-nf">is_removable</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-nb">bool</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">drive_type</span> <span class="z-o">==</span> <span class="z-s1">&#39;Removable Disk&#39;</span> <span class="z-k">class</span> <span class="z-nc">DeviceListener</span><span class="z-p">:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Listens to Win32 `WM_DEVICECHANGE` messages </span><span class="z-s2"> and trigger a callback when a device has been plugged in or out </span><span class="z-s2"> </span><span class="z-s2"> See: https://docs.microsoft.com/en-us/windows/win32/devio/wm-devicechange </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">WM_DEVICECHANGE_EVENTS</span> <span class="z-o">=</span> <span class="z-p">{</span> <span class="z-mh">0x0019</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_CONFIGCHANGECANCELED&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A request to change the current configuration (dock or undock) has been canceled.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x0018</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_CONFIGCHANGED&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;The current configuration has changed, due to a dock or undock.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8006</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_CUSTOMEVENT&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A custom event has occurred.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8000</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEARRIVAL&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A device or piece of media has been inserted and is now available.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8001</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEQUERYREMOVE&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;Permission is requested to remove a device or piece of media. Any application can deny this request and cancel the removal.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8002</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEQUERYREMOVEFAILED&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A request to remove a device or piece of media has been canceled.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8004</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEREMOVECOMPLETE&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A device or piece of media has been removed.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8003</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEREMOVEPENDING&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A device or piece of media is about to be removed. Cannot be denied.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x8005</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICETYPESPECIFIC&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A device-specific event has occurred.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x0007</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVNODES_CHANGED&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;A device has been added to or removed from the system.&#39;</span><span class="z-p">),</span> <span class="z-mh">0x0017</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_QUERYCHANGECONFIG&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;Permission is requested to change the current configuration (dock or undock).&#39;</span><span class="z-p">),</span> <span class="z-mh">0xFFFF</span><span class="z-p">:</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_USERDEFINED&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;The meaning of this message is user-defined.&#39;</span><span class="z-p">),</span> <span class="z-p">}</span> <span class="z-k">def</span> <span class="z-fm">__init__</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">,</span> <span class="z-n">on_change</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]],</span> <span class="z-kc">None</span><span class="z-p">]):</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">on_change</span> <span class="z-o">=</span> <span class="z-n">on_change</span> <span class="z-k">def</span> <span class="z-nf">_create_window</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">):</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Create a window for listening to messages </span><span class="z-s2"> https://docs.microsoft.com/en-us/windows/win32/learnwin32/creating-a-window#creating-the-window </span><span class="z-s2"> </span><span class="z-s2"> See also: https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-createwindoww </span><span class="z-s2"> </span><span class="z-s2"> :return: window hwnd </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">wc</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">WNDCLASS</span><span class="z-p">()</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpfnWndProc</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_on_message</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">lpszClassName</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span> <span class="z-o">=</span> <span class="z-n">win32api</span><span class="z-o">.</span><span class="z-n">GetModuleHandle</span><span class="z-p">(</span><span class="z-kc">None</span><span class="z-p">)</span> <span class="z-n">class_atom</span> <span class="z-o">=</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">RegisterClass</span><span class="z-p">(</span><span class="z-n">wc</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">CreateWindow</span><span class="z-p">(</span><span class="z-n">class_atom</span><span class="z-p">,</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-vm">__class__</span><span class="z-o">.</span><span class="z-vm">__name__</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-mi">0</span><span class="z-p">,</span> <span class="z-n">wc</span><span class="z-o">.</span><span class="z-n">hInstance</span><span class="z-p">,</span> <span class="z-kc">None</span><span class="z-p">)</span> <span class="z-k">def</span> <span class="z-nf">start</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">):</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Listening to drive changes&#39;</span><span class="z-p">)</span> <span class="z-n">hwnd</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">_create_window</span><span class="z-p">()</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">debug</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Created listener window with hwnd=</span><span class="z-si">{</span><span class="z-n">hwnd</span><span class="z-si">:</span><span class="z-s1">x</span><span class="z-si">}</span><span class="z-s1">&#39;</span><span class="z-p">)</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">debug</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Listening to messages&#39;</span><span class="z-p">)</span> <span class="z-n">win32gui</span><span class="z-o">.</span><span class="z-n">PumpMessages</span><span class="z-p">()</span> <span class="z-k">def</span> <span class="z-nf">_on_message</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">,</span> <span class="z-n">hwnd</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">msg</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">wparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">,</span> <span class="z-n">lparam</span><span class="z-p">:</span> <span class="z-nb">int</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-n">msg</span> <span class="z-o">!=</span> <span class="z-n">win32con</span><span class="z-o">.</span><span class="z-n">WM_DEVICECHANGE</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-mi">0</span> <span class="z-n">event</span><span class="z-p">,</span> <span class="z-n">description</span> <span class="z-o">=</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">WM_DEVICECHANGE_EVENTS</span><span class="z-p">[</span><span class="z-n">wparam</span><span class="z-p">]</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">debug</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Received message: </span><span class="z-si">{</span><span class="z-n">event</span><span class="z-si">}</span><span class="z-s1"> = </span><span class="z-si">{</span><span class="z-n">description</span><span class="z-si">}</span><span class="z-s1">&#39;</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">event</span> <span class="z-ow">in</span> <span class="z-p">(</span><span class="z-s1">&#39;DBT_DEVICEREMOVECOMPLETE&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;DBT_DEVICEARRIVAL&#39;</span><span class="z-p">):</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-s1">&#39;A device has been plugged in (or out)&#39;</span><span class="z-p">)</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">on_change</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">list_drives</span><span class="z-p">())</span> <span class="z-k">return</span> <span class="z-mi">0</span> <span class="z-nd">@staticmethod</span> <span class="z-k">def</span> <span class="z-nf">list_drives</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Get a list of drives using WMI </span><span class="z-s2"> :return: list of drives </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">run</span><span class="z-p">(</span> <span class="z-n">args</span><span class="z-o">=</span><span class="z-p">[</span> <span class="z-s1">&#39;powershell&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-noprofile&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-command&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;Get-WmiObject -Class Win32_LogicalDisk | Select-Object deviceid,volumename,drivetype | ConvertTo-Json&#39;</span> <span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span> <span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">returncode</span> <span class="z-o">!=</span> <span class="z-mi">0</span> <span class="z-ow">or</span> <span class="z-ow">not</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">():</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">error</span><span class="z-p">(</span><span class="z-s1">&#39;Failed to enumerate drives&#39;</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-p">[]</span> <span class="z-n">devices</span> <span class="z-o">=</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-p">)</span> <span class="z-n">drive_types</span> <span class="z-o">=</span> <span class="z-p">{</span> <span class="z-mi">0</span><span class="z-p">:</span> <span class="z-s1">&#39;Unknown&#39;</span><span class="z-p">,</span> <span class="z-mi">1</span><span class="z-p">:</span> <span class="z-s1">&#39;No Root Directory&#39;</span><span class="z-p">,</span> <span class="z-mi">2</span><span class="z-p">:</span> <span class="z-s1">&#39;Removable Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">3</span><span class="z-p">:</span> <span class="z-s1">&#39;Local Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">4</span><span class="z-p">:</span> <span class="z-s1">&#39;Network Drive&#39;</span><span class="z-p">,</span> <span class="z-mi">5</span><span class="z-p">:</span> <span class="z-s1">&#39;Compact Disc&#39;</span><span class="z-p">,</span> <span class="z-mi">6</span><span class="z-p">:</span> <span class="z-s1">&#39;RAM Disk&#39;</span><span class="z-p">,</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">(</span> <span class="z-n">letter</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;deviceid&#39;</span><span class="z-p">],</span> <span class="z-n">label</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;volumename&#39;</span><span class="z-p">],</span> <span class="z-n">drive_type</span><span class="z-o">=</span><span class="z-n">drive_types</span><span class="z-p">[</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;drivetype&#39;</span><span class="z-p">]]</span> <span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">d</span> <span class="z-ow">in</span> <span class="z-n">devices</span><span class="z-p">]</span> <span class="z-k">def</span> <span class="z-nf">on_devices_changed</span><span class="z-p">(</span><span class="z-n">drives</span><span class="z-p">:</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]):</span> <span class="z-n">removable_drives</span> <span class="z-o">=</span> <span class="z-p">[</span><span class="z-n">d</span> <span class="z-k">for</span> <span class="z-n">d</span> <span class="z-ow">in</span> <span class="z-n">drives</span> <span class="z-k">if</span> <span class="z-n">d</span><span class="z-o">.</span><span class="z-n">is_removable</span><span class="z-p">]</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">debug</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Connected removable drives: </span><span class="z-si">{</span><span class="z-n">removable_drives</span><span class="z-si">}</span><span class="z-s1">&#39;</span><span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">drive</span> <span class="z-ow">in</span> <span class="z-n">removable_drives</span><span class="z-p">:</span> <span class="z-n">backup</span><span class="z-p">(</span><span class="z-n">drive</span><span class="z-p">)</span> <span class="z-k">def</span> <span class="z-nf">backup</span><span class="z-p">(</span><span class="z-n">drive</span><span class="z-p">:</span> <span class="z-n">Drive</span><span class="z-p">):</span> <span class="z-k">if</span> <span class="z-n">drive</span><span class="z-o">.</span><span class="z-n">label</span> <span class="z-o">!=</span> <span class="z-s1">&#39;ABDUS&#39;</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-s1">&#39;Backup drive has been plugged in&#39;</span><span class="z-p">)</span> <span class="z-n">logger</span><span class="z-o">.</span><span class="z-n">info</span><span class="z-p">(</span><span class="z-sa">f</span><span class="z-s1">&#39;Backing up </span><span class="z-si">{</span><span class="z-n">drive</span><span class="z-o">.</span><span class="z-n">letter</span><span class="z-si">}</span><span class="z-s1">&#39;</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">listener</span> <span class="z-o">=</span> <span class="z-n">DeviceListener</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-o">=</span><span class="z-n">on_devices_changed</span><span class="z-p">)</span> <span class="z-n">listener</span><span class="z-o">.</span><span class="z-n">start</span><span class="z-p">()</span></code></pre> <p>When we run it and plug in a drive, it logs:</p> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">INFO:__main__:Listening to drive changes </span><span class="z-go">DEBUG:__main__:Created listener window with hwnd=d60b86 </span><span class="z-go">DEBUG:__main__:Listening to messages </span><span class="z-go">DEBUG:__main__:Received message: DBT_DEVNODES_CHANGED = A device has been added to or removed from the system. </span><span class="z-go">DEBUG:__main__:Received message: DBT_DEVNODES_CHANGED = A device has been added to or removed from the system. </span><span class="z-go">DEBUG:__main__:Received message: DBT_DEVICEARRIVAL = A device or piece of media has been inserted and is now available. </span><span class="z-go">INFO:__main__:A device has been plugged in (or out) </span><span class="z-go">DEBUG:__main__:Connected removable drives: [Drive(letter=&#39;E:&#39;, label=&#39;ABDUS&#39;, drive_type=&#39;Removable Disk&#39;)] </span><span class="z-go">INFO:__main__:Backup drive has been plugged in </span><span class="z-go">INFO:__main__:Backing up E: </span></code></pre> <p>It works 🙌.</p> <p>Now what you do when your callback is called is up to you. You can:</p> <ul> <li>Copy files to/from the drive</li> <li>Backup photos when an SD card is inserted</li> <li>Display a popup to warn the user</li> <li>Lock the PC</li> <li>Wipe the drive (🤷‍♂️ you do you)</li> <li>...</li> </ul> <h2 id="alternative-approach-polling-for-drives">Alternative approach: polling for drives</h2> <p>This is my older attempt where I poll disks and trigger a callback when the set of drives change. It's considerably simpler, doesn't depend on <a href="https://github.com/mhammond/pywin32" target="_blank" rel="noopener"><code>pywin32</code></a> and easier to port to other operating systems (provided you know how to list drives on that platform).</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">json</span> <span class="z-kn">import</span> <span class="z-nn">subprocess</span> <span class="z-kn">from</span> <span class="z-nn">dataclasses</span> <span class="z-kn">import</span> <span class="z-n">dataclass</span> <span class="z-kn">from</span> <span class="z-nn">typing</span> <span class="z-kn">import</span> <span class="z-n">Callable</span><span class="z-p">,</span> <span class="z-n">List</span> <span class="z-nd">@dataclass</span> <span class="z-k">class</span> <span class="z-nc">Drive</span><span class="z-p">:</span> <span class="z-n">letter</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">label</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-n">drive_type</span><span class="z-p">:</span> <span class="z-nb">str</span> <span class="z-nd">@property</span> <span class="z-k">def</span> <span class="z-nf">is_removable</span><span class="z-p">(</span><span class="z-bp">self</span><span class="z-p">)</span> <span class="z-o">-&gt;</span> <span class="z-nb">bool</span><span class="z-p">:</span> <span class="z-k">return</span> <span class="z-bp">self</span><span class="z-o">.</span><span class="z-n">drive_type</span> <span class="z-o">==</span> <span class="z-s1">&#39;Removable Disk&#39;</span> <span class="z-k">def</span> <span class="z-nf">list_drives</span><span class="z-p">()</span> <span class="z-o">-&gt;</span> <span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]:</span> <span class="z-s2">&#34;&#34;&#34; </span><span class="z-s2"> Get a list of drives using WMI </span><span class="z-s2"> :return: list of drives </span><span class="z-s2"> &#34;&#34;&#34;</span> <span class="z-n">proc</span> <span class="z-o">=</span> <span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">run</span><span class="z-p">(</span> <span class="z-n">args</span><span class="z-o">=</span><span class="z-p">[</span> <span class="z-s1">&#39;powershell&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-noprofile&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;-command&#39;</span><span class="z-p">,</span> <span class="z-s1">&#39;Get-WmiObject -Class Win32_LogicalDisk | Select-Object deviceid,volumename,drivetype | ConvertTo-Json&#39;</span> <span class="z-p">],</span> <span class="z-n">text</span><span class="z-o">=</span><span class="z-kc">True</span><span class="z-p">,</span> <span class="z-n">stdout</span><span class="z-o">=</span><span class="z-n">subprocess</span><span class="z-o">.</span><span class="z-n">PIPE</span> <span class="z-p">)</span> <span class="z-k">if</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">returncode</span> <span class="z-o">!=</span> <span class="z-mi">0</span> <span class="z-ow">or</span> <span class="z-ow">not</span> <span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-o">.</span><span class="z-n">strip</span><span class="z-p">():</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-s1">&#39;Failed to enumerate drives&#39;</span><span class="z-p">)</span> <span class="z-k">return</span> <span class="z-p">[]</span> <span class="z-n">devices</span> <span class="z-o">=</span> <span class="z-n">json</span><span class="z-o">.</span><span class="z-n">loads</span><span class="z-p">(</span><span class="z-n">proc</span><span class="z-o">.</span><span class="z-n">stdout</span><span class="z-p">)</span> <span class="z-n">drive_types</span> <span class="z-o">=</span> <span class="z-p">{</span> <span class="z-mi">0</span><span class="z-p">:</span> <span class="z-s1">&#39;Unknown&#39;</span><span class="z-p">,</span> <span class="z-mi">1</span><span class="z-p">:</span> <span class="z-s1">&#39;No Root Directory&#39;</span><span class="z-p">,</span> <span class="z-mi">2</span><span class="z-p">:</span> <span class="z-s1">&#39;Removable Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">3</span><span class="z-p">:</span> <span class="z-s1">&#39;Local Disk&#39;</span><span class="z-p">,</span> <span class="z-mi">4</span><span class="z-p">:</span> <span class="z-s1">&#39;Network Drive&#39;</span><span class="z-p">,</span> <span class="z-mi">5</span><span class="z-p">:</span> <span class="z-s1">&#39;Compact Disc&#39;</span><span class="z-p">,</span> <span class="z-mi">6</span><span class="z-p">:</span> <span class="z-s1">&#39;RAM Disk&#39;</span><span class="z-p">,</span> <span class="z-p">}</span> <span class="z-k">return</span> <span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">(</span> <span class="z-n">letter</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;deviceid&#39;</span><span class="z-p">],</span> <span class="z-n">label</span><span class="z-o">=</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;volumename&#39;</span><span class="z-p">],</span> <span class="z-n">drive_type</span><span class="z-o">=</span><span class="z-n">drive_types</span><span class="z-p">[</span><span class="z-n">d</span><span class="z-p">[</span><span class="z-s1">&#39;drivetype&#39;</span><span class="z-p">]]</span> <span class="z-p">)</span> <span class="z-k">for</span> <span class="z-n">d</span> <span class="z-ow">in</span> <span class="z-n">devices</span><span class="z-p">]</span> <span class="z-k">def</span> <span class="z-nf">watch_drives</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-p">:</span> <span class="z-n">Callable</span><span class="z-p">[[</span><span class="z-n">List</span><span class="z-p">[</span><span class="z-n">Drive</span><span class="z-p">]],</span> <span class="z-kc">None</span><span class="z-p">],</span> <span class="z-n">poll_interval</span><span class="z-p">:</span> <span class="z-nb">int</span> <span class="z-o">=</span> <span class="z-mi">1</span><span class="z-p">):</span> <span class="z-n">prev</span> <span class="z-o">=</span> <span class="z-kc">None</span> <span class="z-k">while</span> <span class="z-kc">True</span><span class="z-p">:</span> <span class="z-n">drives</span> <span class="z-o">=</span> <span class="z-n">list_drives</span><span class="z-p">()</span> <span class="z-k">if</span> <span class="z-n">prev</span> <span class="z-o">!=</span> <span class="z-n">drives</span><span class="z-p">:</span> <span class="z-n">on_change</span><span class="z-p">(</span><span class="z-n">drives</span><span class="z-p">)</span> <span class="z-n">prev</span> <span class="z-o">=</span> <span class="z-n">drives</span> <span class="z-n">sleep</span><span class="z-p">(</span><span class="z-n">poll_interval</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-vm">__name__</span> <span class="z-o">==</span> <span class="z-s1">&#39;__main__&#39;</span><span class="z-p">:</span> <span class="z-n">watch_drives</span><span class="z-p">(</span><span class="z-n">on_change</span><span class="z-o">=</span><span class="z-nb">print</span><span class="z-p">)</span></code></pre> <p>You can supply your own <code>on_change</code> callback and perform a job if a drive is present.</p> <p>Here's the output after I plug a USB drive in and out:</p> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">[Drive(letter=&#39;C:&#39;, label=&#39;&#39;, drive_type=&#39;Local Disk&#39;)] </span><span class="z-go">[Drive(letter=&#39;C:&#39;, label=&#39;&#39;, drive_type=&#39;Local Disk&#39;), Drive(letter=&#39;E:&#39;, label=&#39;ABDUS&#39;, drive_type=&#39;Removable Disk&#39;)] </span><span class="z-go">[Drive(letter=&#39;C:&#39;, label=&#39;&#39;, drive_type=&#39;Local Disk&#39;)] </span></code></pre> <p>That's it.<br /> Cheers ✌</p> <p>If you've found this post useful, consider sharing it.</p> Tue, 16 Jun 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/powershell-hotspot/ https://abdus.dev/posts/powershell-hotspot/ <p>There used to be a way to turn on or off the hotspot on Windows 7, but it doesn't seem to work well with some wifi adapters like Intel 9260 wifi card I've installed on my XPS 13. So I cannot control hotspot using <code>netsh</code>.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-n">PS</span><span class="z-p">&gt;</span> <span class="z-n">netsh</span> <span class="z-n">wlan</span> <span class="z-nb">start </span><span class="z-n">hostednetwork</span> <span class="z-n">The</span> <span class="z-n">hosted</span> <span class="z-n">network</span> <span class="z-n">couldn</span><span class="z-err">&#39;</span><span class="z-n">t</span> <span class="z-n">be</span> <span class="z-n">started</span><span class="z-p">.</span> <span class="z-n">The</span> <span class="z-nb">group </span><span class="z-n">or</span> <span class="z-n">resource</span> <span class="z-n">is</span> <span class="z-n">not</span> <span class="z-k">in</span> <span class="z-n">the</span> <span class="z-n">correct</span> <span class="z-n">state</span> <span class="z-n">to</span> <span class="z-n">perform</span> <span class="z-n">the</span> <span class="z-n">requested</span> <span class="z-n">operation</span><span class="z-p">.</span></code></pre> <p>The only way to control native hotspot was so far using the UI</p> <p><img src="/posts/powershell-hotspot/tray.png" alt="" /></p> <p>As of Windows 10 v1607 there's a <a href="https://docs.microsoft.com/en-us/uwp/api/windows.networking.networkoperators.networkoperatortetheringmanager" target="_blank" rel="noopener">new API</a> for UWP apps that we can use to start / stop hotspot tethering in PowerShell:</p> <p>Here's the script based off of <a href="https://stackoverflow.com/a/55563418/5298150" target="_blank" rel="noopener">this answer</a> on StackOverflow:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-k">param</span> <span class="z-p">(</span> <span class="z-p">[</span><span class="z-no">string</span><span class="z-p">]</span> <span class="z-nv">$state</span> <span class="z-p">)</span> <span class="z-nv">$connectionProfile</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-n">Windows</span><span class="z-p">.</span><span class="z-py">Networking</span><span class="z-p">.</span><span class="z-py">Connectivity</span><span class="z-p">.</span><span class="z-n">NetworkInformation</span><span class="z-p">,</span> <span class="z-n">Windows</span><span class="z-p">.</span><span class="z-py">Networking</span><span class="z-p">.</span><span class="z-n">Connectivity</span><span class="z-p">,</span> <span class="z-n">ContentType</span> <span class="z-p">=</span> <span class="z-n">WindowsRuntime</span><span class="z-p">]::</span><span class="z-n">GetInternetConnectionProfile</span><span class="z-p">()</span> <span class="z-nv">$tetheringManager</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-n">Windows</span><span class="z-p">.</span><span class="z-py">Networking</span><span class="z-p">.</span><span class="z-py">NetworkOperators</span><span class="z-p">.</span><span class="z-n">NetworkOperatorTetheringManager</span><span class="z-p">,</span> <span class="z-n">Windows</span><span class="z-p">.</span><span class="z-py">Networking</span><span class="z-p">.</span><span class="z-n">NetworkOperators</span><span class="z-p">,</span> <span class="z-n">ContentType</span> <span class="z-p">=</span> <span class="z-n">WindowsRuntime</span><span class="z-p">]::</span><span class="z-n">CreateFromConnectionProfile</span><span class="z-p">(</span><span class="z-nv">$connectionProfile</span><span class="z-p">)</span> <span class="z-k">if</span> <span class="z-p">(</span><span class="z-nv">$state</span> <span class="z-o">-eq</span> <span class="z-s2">&#34;on&#34;</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nv">$tetheringManager</span><span class="z-p">.</span><span class="z-py">StartTetheringAsync</span><span class="z-p">()</span> <span class="z-p">|</span> <span class="z-nb">Out-Null</span> <span class="z-p">}</span> <span class="z-k">elseif</span> <span class="z-p">(</span><span class="z-nv">$state</span> <span class="z-o">-eq</span> <span class="z-s2">&#34;off&#34;</span><span class="z-p">)</span> <span class="z-p">{</span> <span class="z-nv">$tetheringManager</span><span class="z-p">.</span><span class="z-py">StopTetheringAsync</span><span class="z-p">()</span> <span class="z-p">|</span> <span class="z-nb">Out-Null</span> <span class="z-p">}</span> <span class="z-k">else</span> <span class="z-p">{</span> <span class="z-nv">$tetheringManager</span><span class="z-p">.</span><span class="z-py">TetheringOperationalState</span> <span class="z-p">}</span></code></pre> <p>Save this as <code>hotspot.ps1</code> and run it from PowerShell</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-p">.\</span><span class="z-n">hotspot</span><span class="z-p">.</span><span class="z-py">ps1</span> <span class="z-c"># get the current state (on/off)</span> <span class="z-p">.\</span><span class="z-n">hotspot</span><span class="z-p">.</span><span class="z-py">ps1</span> <span class="z-n">on</span> <span class="z-p">.\</span><span class="z-n">hotspot</span><span class="z-p">.</span><span class="z-py">ps1</span> <span class="z-n">off</span></code></pre> <p>or call it from cmd:</p> <pre data-lang="cmd" data-highlighted="true" class="language-cmd z-code z-chroma"><code class="language-cmd" data-lang="cmd">powershell -file hotspot.ps1 on </code></pre> Mon, 25 May 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/powershell-file-metadata-guide/ https://abdus.dev/posts/powershell-file-metadata-guide/ <div class="table-of-contents"> <ul> <li style="margin-left: 0rem"><a href="#common-metadata">Common metadata</a></li> <li style="margin-left: 1rem"><a href="#creation-and-modification-date">Creation and modification date</a></li> <li style="margin-left: 0rem"><a href="#media-files">Media files</a></li> <li style="margin-left: 1rem"><a href="#loading-taglib">Loading TagLib#</a></li> <li style="margin-left: 1rem"><a href="#image-metadata">Image metadata</a></li> <li style="margin-left: 1rem"><a href="#video-metadata">Video metadata</a></li> <li style="margin-left: 1rem"><a href="#cover-photo">Cover photo</a></li> </ul> </div> <h2 id="common-metadata">Common metadata</h2> <h3 id="creation-and-modification-date">Creation and modification date</h3> <ul> <li> <p>Single file:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-p">(</span><span class="z-nb">Get-ChildItem</span> <span class="z-p">.\</span><span class="z-n">file</span><span class="z-p">.</span><span class="z-n">ext</span><span class="z-p">).</span><span class="z-py">CreationTime</span> <span class="z-p">=</span> <span class="z-s2">&#34;2026-10-07 12:00&#34;</span> <span class="z-p">(</span><span class="z-nb">Get-ChildItem</span> <span class="z-p">.\</span><span class="z-n">file</span><span class="z-p">.</span><span class="z-n">ext</span><span class="z-p">).</span><span class="z-py">LastWriteTime</span> <span class="z-p">=</span> <span class="z-s2">&#34;2026-10-07 12:00&#34;</span></code></pre> </li> <li> <p>All files in current folder:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nb">Get-ChildItem</span> <span class="z-p">|</span> <span class="z-nb">ForEach-Object</span> <span class="z-p">{</span> <span class="z-nv">$_</span><span class="z-p">.</span><span class="z-py">CreationTime</span> <span class="z-p">=</span> <span class="z-s2">&#34;2026-10-07&#34;</span> <span class="z-p">}</span> <span class="z-nb">Get-ChildItem</span> <span class="z-p">|</span> <span class="z-nb">ForEach-Object</span> <span class="z-p">{</span> <span class="z-nv">$_</span><span class="z-p">.</span><span class="z-py">LastWriteTime</span> <span class="z-p">=</span> <span class="z-s2">&#34;2026-10-07&#34;</span> <span class="z-p">}</span></code></pre> </li> </ul> <h2 id="media-files">Media files</h2> <h3 id="loading-taglib">Loading TagLib#</h3> <p><a href="https://github.com/mono/taglib-sharp" target="_blank" rel="noopener">TagLib#</a> is a really great library crafted for this job. Download the <a href="https://www.nuget.org/api/v2/package/TagLibSharp" target="_blank" rel="noopener">NuGet package</a> and unzip it. Copy the <code>TagLibSharp.dll</code> into where you'll be running the script.</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell">. ├── lib │ ├── net45 │ │ ├── ... │ └── netstandard2.0 │ ├── TagLibSharp.dll &lt;-- we need this one │ ├── TagLibSharp.pdb │ └── TaglibSharp.xml ├── ... └── ...</code></pre> <p>Load the DLL:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-p">[</span><span class="z-no">System.Reflection.Assembly</span><span class="z-p">]::</span><span class="z-n">LoadFrom</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-s2">&#34;TagLibSharp.dll&#34;</span><span class="z-p">))</span></code></pre> <h3 id="image-metadata">Image metadata</h3> <p>Open the file and set properties in <code>.ImageTag</code> object and save it.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nv">$photo</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">TagLib.File</span><span class="z-p">]::</span><span class="z-n">Create</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-s2">&#34;photo.jpg&#34;</span><span class="z-p">))</span> <span class="z-nv">$photo</span><span class="z-p">.</span><span class="z-py">ImageTag</span><span class="z-p">.</span><span class="z-py">Title</span> <span class="z-p">=</span> <span class="z-s2">&#34;hello&#34;</span> <span class="z-nv">$photo</span><span class="z-p">.</span><span class="z-py">ImageTag</span><span class="z-p">.</span><span class="z-py">Keywords</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-s2">&#34;tag&#34;</span><span class="z-p">,</span> <span class="z-s2">&#34;another tag&#34;</span><span class="z-p">)</span> <span class="z-nv">$photo</span><span class="z-p">.</span><span class="z-py">ImageTag</span><span class="z-p">.</span><span class="z-py">Comment</span> <span class="z-p">=</span> <span class="z-s2">&#34;my comment&#34;</span> <span class="z-nv">$photo</span><span class="z-p">.</span><span class="z-py">ImageTag</span><span class="z-p">.</span><span class="z-py">DateTime</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">System.DateTime</span><span class="z-p">]::</span><span class="z-n">Parse</span><span class="z-p">(</span><span class="z-s2">&#34;2020-01-02&#34;</span><span class="z-p">)</span> <span class="z-nv">$photo</span><span class="z-p">.</span><span class="z-py">Save</span><span class="z-p">()</span></code></pre> <h3 id="video-metadata">Video metadata</h3> <p><a href="https://github.com/mono/taglib-sharp" target="_blank" rel="noopener">TagLibSharp</a> can modify video metadata too.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nv">$video</span> <span class="z-p">=</span> <span class="z-nv">$video</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">TagLib.File</span><span class="z-p">]::</span><span class="z-n">Create</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-p">./</span><span class="z-n">video</span><span class="z-p">.</span><span class="z-n">mkv</span><span class="z-p">))</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Title</span> <span class="z-p">=</span> <span class="z-s2">&#34;my video&#34;</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Year</span> <span class="z-p">=</span> <span class="z-mf">2020</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Comment</span> <span class="z-p">=</span> <span class="z-s2">&#34;a comment&#34;</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Publisher</span> <span class="z-p">=</span> <span class="z-s2">&#34;a publisher&#34;</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Description</span> <span class="z-p">=</span> <span class="z-s2">&#34;a description&#34;</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Save</span><span class="z-p">()</span></code></pre> <h3 id="cover-photo">Cover photo</h3> <p>Load the image file and override <code>.Tag.Pictures</code> with an array.</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nv">$cover</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">TagLib.Picture</span><span class="z-p">]::</span><span class="z-n">CreateFromPath</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-n">cover</span><span class="z-p">.</span><span class="z-n">png</span><span class="z-p">))</span> <span class="z-nv">$video</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">TagLib.File</span><span class="z-p">]::</span><span class="z-n">Create</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-n">video</span><span class="z-p">.</span><span class="z-n">mp4</span><span class="z-p">))</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Pictures</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-nv">$cover</span><span class="z-p">)</span> <span class="z-nv">$video</span><span class="z-p">.</span><span class="z-py">Save</span><span class="z-p">()</span> <span class="z-nv">$music</span> <span class="z-p">=</span> <span class="z-p">[</span><span class="z-no">TagLib.File</span><span class="z-p">]::</span><span class="z-n">Create</span><span class="z-p">((</span><span class="z-nb">Resolve-Path</span> <span class="z-n">music</span><span class="z-p">.</span><span class="z-n">mp3</span><span class="z-p">))</span> <span class="z-nv">$music</span><span class="z-p">.</span><span class="z-py">Tag</span><span class="z-p">.</span><span class="z-py">Pictures</span> <span class="z-p">=</span> <span class="z-p">(</span><span class="z-nv">$cover</span><span class="z-p">)</span> <span class="z-nv">$music</span><span class="z-p">.</span><span class="z-py">Save</span><span class="z-p">()</span></code></pre> Mon, 18 May 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/powershell-aliases/ https://abdus.dev/posts/powershell-aliases/ <p>A list of current aliases can be retrieved using <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/get-alias" target="_blank" rel="noopener"><code>Get-Alias</code></a> command.</p> <table> <thead> <tr> <th>Alias</th> <th>Command</th> </tr> </thead> <tbody> <tr> <td><code>%</code></td> <td><code>ForEach-Object</code></td> </tr> <tr> <td><code>?</code></td> <td><code>Where-Object</code></td> </tr> <tr> <td><code>ac</code></td> <td><code>Add-Content</code></td> </tr> <tr> <td><code>asnp</code></td> <td><code>Add-PSSnapin</code></td> </tr> <tr> <td><code>cat</code></td> <td><code>Get-Content</code></td> </tr> <tr> <td><code>cd</code></td> <td><code>Set-Location</code></td> </tr> <tr> <td><code>CFS</code></td> <td><code>ConvertFrom-String</code></td> </tr> <tr> <td><code>chdir</code></td> <td><code>Set-Location</code></td> </tr> <tr> <td><code>clc</code></td> <td><code>Clear-Content</code></td> </tr> <tr> <td><code>clear</code></td> <td><code>Clear-Host</code></td> </tr> <tr> <td><code>clhy</code></td> <td><code>Clear-History</code></td> </tr> <tr> <td><code>cli</code></td> <td><code>Clear-Item</code></td> </tr> <tr> <td><code>clp</code></td> <td><code>Clear-ItemProperty</code></td> </tr> <tr> <td><code>cls</code></td> <td><code>Clear-Host</code></td> </tr> <tr> <td><code>clv</code></td> <td><code>Clear-Variable</code></td> </tr> <tr> <td><code>cnsn</code></td> <td><code>Connect-PSSession</code></td> </tr> <tr> <td><code>compare</code></td> <td><code>Compare-Object</code></td> </tr> <tr> <td><code>copy</code></td> <td><code>Copy-Item</code></td> </tr> <tr> <td><code>cp</code></td> <td><code>Copy-Item</code></td> </tr> <tr> <td><code>cpi</code></td> <td><code>Copy-Item</code></td> </tr> <tr> <td><code>cpp</code></td> <td><code>Copy-ItemProperty</code></td> </tr> <tr> <td><code>curl</code></td> <td><code>Invoke-WebRequest</code></td> </tr> <tr> <td><code>cvpa</code></td> <td><code>Convert-Path</code></td> </tr> <tr> <td><code>dbp</code></td> <td><code>Disable-PSBreakpoint</code></td> </tr> <tr> <td><code>del</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>diff</code></td> <td><code>Compare-Object</code></td> </tr> <tr> <td><code>dir</code></td> <td><code>Get-ChildItem</code></td> </tr> <tr> <td><code>dnsn</code></td> <td><code>Disconnect-PSSession</code></td> </tr> <tr> <td><code>ebp</code></td> <td><code>Enable-PSBreakpoint</code></td> </tr> <tr> <td><code>echo</code></td> <td><code>Write-Output</code></td> </tr> <tr> <td><code>epal</code></td> <td><code>Export-Alias</code></td> </tr> <tr> <td><code>epcsv</code></td> <td><code>Export-Csv</code></td> </tr> <tr> <td><code>epsn</code></td> <td><code>Export-PSSession</code></td> </tr> <tr> <td><code>erase</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>etsn</code></td> <td><code>Enter-PSSession</code></td> </tr> <tr> <td><code>exsn</code></td> <td><code>Exit-PSSession</code></td> </tr> <tr> <td><code>fc</code></td> <td><code>Format-Custom</code></td> </tr> <tr> <td><code>fhx</code></td> <td><code>Format-Hex</code></td> </tr> <tr> <td><code>fl</code></td> <td><code>Format-List</code></td> </tr> <tr> <td><code>foreach</code></td> <td><code>ForEach-Object</code></td> </tr> <tr> <td><code>ft</code></td> <td><code>Format-Table</code></td> </tr> <tr> <td><code>fw</code></td> <td><code>Format-Wide</code></td> </tr> <tr> <td><code>gal</code></td> <td><code>Get-Alias</code></td> </tr> <tr> <td><code>gbp</code></td> <td><code>Get-PSBreakpoint</code></td> </tr> <tr> <td><code>gc</code></td> <td><code>Get-Content</code></td> </tr> <tr> <td><code>gcb</code></td> <td><code>Get-Clipboard</code></td> </tr> <tr> <td><code>gci</code></td> <td><code>Get-ChildItem</code></td> </tr> <tr> <td><code>gcm</code></td> <td><code>Get-Command</code></td> </tr> <tr> <td><code>gcs</code></td> <td><code>Get-PSCallStack</code></td> </tr> <tr> <td><code>gdr</code></td> <td><code>Get-PSDrive</code></td> </tr> <tr> <td><code>ghy</code></td> <td><code>Get-History</code></td> </tr> <tr> <td><code>gi</code></td> <td><code>Get-Item</code></td> </tr> <tr> <td><code>gin</code></td> <td><code>Get-ComputerInfo</code></td> </tr> <tr> <td><code>gjb</code></td> <td><code>Get-Job</code></td> </tr> <tr> <td><code>gl</code></td> <td><code>Get-Location</code></td> </tr> <tr> <td><code>gm</code></td> <td><code>Get-Member</code></td> </tr> <tr> <td><code>gmo</code></td> <td><code>Get-Module</code></td> </tr> <tr> <td><code>gp</code></td> <td><code>Get-ItemProperty</code></td> </tr> <tr> <td><code>gps</code></td> <td><code>Get-Process</code></td> </tr> <tr> <td><code>gpv</code></td> <td><code>Get-ItemPropertyValue</code></td> </tr> <tr> <td><code>group</code></td> <td><code>Group-Object</code></td> </tr> <tr> <td><code>gsn</code></td> <td><code>Get-PSSession</code></td> </tr> <tr> <td><code>gsnp</code></td> <td><code>Get-PSSnapin</code></td> </tr> <tr> <td><code>gsv</code></td> <td><code>Get-Service</code></td> </tr> <tr> <td><code>gtz</code></td> <td><code>Get-TimeZone</code></td> </tr> <tr> <td><code>gu</code></td> <td><code>Get-Unique</code></td> </tr> <tr> <td><code>gv</code></td> <td><code>Get-Variable</code></td> </tr> <tr> <td><code>gwmi</code></td> <td><code>Get-WmiObject</code></td> </tr> <tr> <td><code>h</code></td> <td><code>Get-History</code></td> </tr> <tr> <td><code>history</code></td> <td><code>Get-History</code></td> </tr> <tr> <td><code>icm</code></td> <td><code>Invoke-Command</code></td> </tr> <tr> <td><code>iex</code></td> <td><code>Invoke-Expression</code></td> </tr> <tr> <td><code>ihy</code></td> <td><code>Invoke-History</code></td> </tr> <tr> <td><code>ii</code></td> <td><code>Invoke-Item</code></td> </tr> <tr> <td><code>ipal</code></td> <td><code>Import-Alias</code></td> </tr> <tr> <td><code>ipcsv</code></td> <td><code>Import-Csv</code></td> </tr> <tr> <td><code>ipmo</code></td> <td><code>Import-Module</code></td> </tr> <tr> <td><code>ipsn</code></td> <td><code>Import-PSSession</code></td> </tr> <tr> <td><code>irm</code></td> <td><code>Invoke-RestMethod</code></td> </tr> <tr> <td><code>ise</code></td> <td><code>powershell_ise.exe</code></td> </tr> <tr> <td><code>iwmi</code></td> <td><code>Invoke-WmiMethod</code></td> </tr> <tr> <td><code>iwr</code></td> <td><code>Invoke-WebRequest</code></td> </tr> <tr> <td><code>kill</code></td> <td><code>Stop-Process</code></td> </tr> <tr> <td><code>lp</code></td> <td><code>Out-Printer</code></td> </tr> <tr> <td><code>ls</code></td> <td><code>Get-ChildItem</code></td> </tr> <tr> <td><code>man</code></td> <td><code>help</code></td> </tr> <tr> <td><code>md</code></td> <td><code>mkdir</code></td> </tr> <tr> <td><code>measure</code></td> <td><code>Measure-Object</code></td> </tr> <tr> <td><code>mi</code></td> <td><code>Move-Item</code></td> </tr> <tr> <td><code>mount</code></td> <td><code>New-PSDrive</code></td> </tr> <tr> <td><code>move</code></td> <td><code>Move-Item</code></td> </tr> <tr> <td><code>mp</code></td> <td><code>Move-ItemProperty</code></td> </tr> <tr> <td><code>mv</code></td> <td><code>Move-Item</code></td> </tr> <tr> <td><code>nal</code></td> <td><code>New-Alias</code></td> </tr> <tr> <td><code>ndr</code></td> <td><code>New-PSDrive</code></td> </tr> <tr> <td><code>ni</code></td> <td><code>New-Item</code></td> </tr> <tr> <td><code>nmo</code></td> <td><code>New-Module</code></td> </tr> <tr> <td><code>npssc</code></td> <td><code>New-PSSessionConfigurationFile</code></td> </tr> <tr> <td><code>nsn</code></td> <td><code>New-PSSession</code></td> </tr> <tr> <td><code>nv</code></td> <td><code>New-Variable</code></td> </tr> <tr> <td><code>ogv</code></td> <td><code>Out-GridView</code></td> </tr> <tr> <td><code>oh</code></td> <td><code>Out-Host</code></td> </tr> <tr> <td><code>popd</code></td> <td><code>Pop-Location</code></td> </tr> <tr> <td><code>ps</code></td> <td><code>Get-Process</code></td> </tr> <tr> <td><code>pushd</code></td> <td><code>Push-Location</code></td> </tr> <tr> <td><code>pwd</code></td> <td><code>Get-Location</code></td> </tr> <tr> <td><code>r</code></td> <td><code>Invoke-History</code></td> </tr> <tr> <td><code>rbp</code></td> <td><code>Remove-PSBreakpoint</code></td> </tr> <tr> <td><code>rcjb</code></td> <td><code>Receive-Job</code></td> </tr> <tr> <td><code>rcsn</code></td> <td><code>Receive-PSSession</code></td> </tr> <tr> <td><code>rd</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>rdr</code></td> <td><code>Remove-PSDrive</code></td> </tr> <tr> <td><code>ren</code></td> <td><code>Rename-Item</code></td> </tr> <tr> <td><code>ri</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>rjb</code></td> <td><code>Remove-Job</code></td> </tr> <tr> <td><code>rm</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>rmdir</code></td> <td><code>Remove-Item</code></td> </tr> <tr> <td><code>rmo</code></td> <td><code>Remove-Module</code></td> </tr> <tr> <td><code>rni</code></td> <td><code>Rename-Item</code></td> </tr> <tr> <td><code>rnp</code></td> <td><code>Rename-ItemProperty</code></td> </tr> <tr> <td><code>rp</code></td> <td><code>Remove-ItemProperty</code></td> </tr> <tr> <td><code>rsn</code></td> <td><code>Remove-PSSession</code></td> </tr> <tr> <td><code>rsnp</code></td> <td><code>Remove-PSSnapin</code></td> </tr> <tr> <td><code>rujb</code></td> <td><code>Resume-Job</code></td> </tr> <tr> <td><code>rv</code></td> <td><code>Remove-Variable</code></td> </tr> <tr> <td><code>rvpa</code></td> <td><code>Resolve-Path</code></td> </tr> <tr> <td><code>rwmi</code></td> <td><code>Remove-WmiObject</code></td> </tr> <tr> <td><code>sajb</code></td> <td><code>Start-Job</code></td> </tr> <tr> <td><code>sal</code></td> <td><code>Set-Alias</code></td> </tr> <tr> <td><code>saps</code></td> <td><code>Start-Process</code></td> </tr> <tr> <td><code>sasv</code></td> <td><code>Start-Service</code></td> </tr> <tr> <td><code>sbp</code></td> <td><code>Set-PSBreakpoint</code></td> </tr> <tr> <td><code>sc</code></td> <td><code>Set-Content</code></td> </tr> <tr> <td><code>scb</code></td> <td><code>Set-Clipboard</code></td> </tr> <tr> <td><code>select</code></td> <td><code>Select-Object</code></td> </tr> <tr> <td><code>set</code></td> <td><code>Set-Variable</code></td> </tr> <tr> <td><code>shcm</code></td> <td><code>Show-Command</code></td> </tr> <tr> <td><code>si</code></td> <td><code>Set-Item</code></td> </tr> <tr> <td><code>sl</code></td> <td><code>Set-Location</code></td> </tr> <tr> <td><code>sleep</code></td> <td><code>Start-Sleep</code></td> </tr> <tr> <td><code>sls</code></td> <td><code>Select-String</code></td> </tr> <tr> <td><code>sort</code></td> <td><code>Sort-Object</code></td> </tr> <tr> <td><code>sp</code></td> <td><code>Set-ItemProperty</code></td> </tr> <tr> <td><code>spjb</code></td> <td><code>Stop-Job</code></td> </tr> <tr> <td><code>spps</code></td> <td><code>Stop-Process</code></td> </tr> <tr> <td><code>spsv</code></td> <td><code>Stop-Service</code></td> </tr> <tr> <td><code>start</code></td> <td><code>Start-Process</code></td> </tr> <tr> <td><code>stz</code></td> <td><code>Set-TimeZone</code></td> </tr> <tr> <td><code>sujb</code></td> <td><code>Suspend-Job</code></td> </tr> <tr> <td><code>sv</code></td> <td><code>Set-Variable</code></td> </tr> <tr> <td><code>swmi</code></td> <td><code>Set-WmiInstance</code></td> </tr> <tr> <td><code>tee</code></td> <td><code>Tee-Object</code></td> </tr> <tr> <td><code>trcm</code></td> <td><code>Trace-Command</code></td> </tr> <tr> <td><code>type</code></td> <td><code>Get-Content</code></td> </tr> <tr> <td><code>wget</code></td> <td><code>Invoke-WebRequest</code></td> </tr> <tr> <td><code>where</code></td> <td><code>Where-Object</code></td> </tr> <tr> <td><code>wjb</code></td> <td><code>Wait-Job</code></td> </tr> <tr> <td><code>write</code></td> <td><code>Write-Output</code></td> </tr> </tbody> </table> Thu, 20 Feb 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/sqlite-upgrade-windows/ https://abdus.dev/posts/sqlite-upgrade-windows/ <p>SQLite has an extension called <a href="https://www.sqlite.org/json1.html" target="_blank" rel="noopener"><code>JSON1</code></a> that allows working with JSON. It's really useful for storing data that wouldn't necessarily belong to a separate column, like metadata.</p> <p>Unfortunately on Windows, SQLite driver for Python doesn't come bundled with this extension. We need to upgrade the driver to a newer version to get it.</p> <p>You can find the compiled binaries for Windows on <a href="https://www.sqlite.org/download.html#win32" target="_blank" rel="noopener">SQLite website</a>. Here are the links, scraped directly from the page:</p> <div class="download"> <ul id="sqlite-downloads"> <li>Fetching the latest download URLs...</li> </ul> <script> (async () => { const list = document.getElementById('sqlite-downloads'); try { const url = 'https://www.sqlite.org/download.html#win32'; const response = await fetch(`https://cors.abdus.dev?url=${encodeURIComponent(url)}`); if (!response.ok) throw new Error('Failed to fetch SQLite downloads'); const html = await response.text(); const paths = [...new Set((html.match(/\d+\/[^\s"'<>]+\.zip/g) || []) .filter(path => /dll-win/.test(path)))]; if (!paths.length) throw new Error('No SQLite DLL downloads found'); list.replaceChildren(...paths.map(path => { const item = document.createElement('li'); const link = document.createElement('a'); link.href = `https://www.sqlite.org/${path}`; link.textContent = link.href; item.append(link); return item; })); } catch (error) { list.firstElementChild.textContent = 'Unable to fetch downloads. Please use the SQLite website link above.'; } })(); </script> </div> <p>Extract the ZIP file and place the DLL into <code>$PYTHON_DIR/DLLs</code>.</p> <p>You can find the exact location using PowerShell:</p> <pre data-lang="powershell" data-highlighted="true" class="language-powershell z-code z-chroma"><code class="language-powershell" data-lang="powershell"><span class="z-nb">start </span><span class="z-p">(</span><span class="z-nb">Join-Path</span> <span class="z-p">(</span><span class="z-nb">Split-Path</span> <span class="z-p">(</span><span class="z-nb">Get-Command</span> <span class="z-n">python</span><span class="z-p">).</span><span class="z-n">Path</span><span class="z-p">)</span> <span class="z-s2">&#34;dlls&#34;</span><span class="z-p">)</span> <span class="z-c"># C:\Python39\dlls</span></code></pre> <p>or use cmd directly:</p> <pre data-lang="cmd" data-highlighted="true" class="language-cmd z-code z-chroma"><code class="language-cmd" data-lang="cmd">powershell -command <span class="z-s2">&#34;start (Join-Path (Split-Path (Get-Command python).Path) &#34;</span>dlls<span class="z-s2">&#34;)&#34;</span></code></pre> <h2 id="sqlite-compile-options">SQLite compile options</h2> <p>Run <code>python</code>, and fetch the list of compilation options to verify if it has succeeded:</p> <pre data-lang="python" data-highlighted="true" class="language-python z-code z-chroma"><code class="language-python" data-lang="python"><span class="z-kn">import</span> <span class="z-nn">sqlite3</span> <span class="z-n">options</span> <span class="z-o">=</span> <span class="z-n">sqlite3</span><span class="z-o">.</span><span class="z-n">connect</span><span class="z-p">(</span><span class="z-s1">&#39;:memory:&#39;</span><span class="z-p">)</span><span class="z-o">.</span><span class="z-n">execute</span><span class="z-p">(</span><span class="z-s1">&#39;pragma compile_options&#39;</span><span class="z-p">)</span><span class="z-o">.</span><span class="z-n">fetchall</span><span class="z-p">()</span> <span class="z-k">for</span> <span class="z-n">o</span> <span class="z-ow">in</span> <span class="z-n">options</span><span class="z-p">:</span> <span class="z-nb">print</span><span class="z-p">(</span><span class="z-n">o</span><span class="z-p">[</span><span class="z-mi">0</span><span class="z-p">])</span></code></pre> <pre data-lang="console" data-highlighted="true" class="language-console z-code z-chroma"><code class="language-console" data-lang="console"><span class="z-go">COMPILER=msvc-1500 </span><span class="z-go">ENABLE_BYTECODE_VTAB </span><span class="z-go">ENABLE_COLUMN_METADATA </span><span class="z-go">ENABLE_DBSTAT_VTAB </span><span class="z-go">ENABLE_FTS3 </span><span class="z-go">ENABLE_FTS4 </span><span class="z-go">ENABLE_FTS5 </span><span class="z-go">ENABLE_GEOPOLY </span><span class="z-go">ENABLE_JSON1 &lt;-- IT WORKS! 🎉 </span><span class="z-go">ENABLE_RTREE </span><span class="z-go">ENABLE_STMTVTAB </span><span class="z-go">MAX_TRIGGER_DEPTH=100 </span><span class="z-go">TEMP_STORE=1 </span><span class="z-go">THREADSAFE=1 </span></code></pre> <p>Thanks to the new driver, we also have access to <a href="https://www.sqlite.org/fts5.html" target="_blank" rel="noopener"><code>FTS5</code></a> extension to work with full text search indices.</p> Mon, 20 Jan 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/prayertimes/ https://abdus.dev/projects/prayertimes/ <p><strong>2026 EDIT</strong>: ⚠️ This API has been deprecated, please use <a href="/projects/prayertimes-v2/">the new version</a> instead.</p> <hr /> <p>This API lets you fetch daily prayer times from <a href="https://namazvakitleri.diyanet.gov.tr/en-US" target="_blank" rel="noopener">Presidency of Religious Affairs of Turkey</a> (Diyanet İşleri Başkanlığı) in a nice, clean format. You can use it to find a location with available prayer times info, then query the schedule of that month's prayer times.</p> <a class="repo-card" href="https://git.abdus.dev/abdus/prayertimes" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdus</span> <span aria-hidden="true">/</span> <strong>prayertimes</strong> </span> <span class="repo-card__meta"> <span>Python</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> <p>I've created this API to prepare for a web app I'll be building soon. I am planning to structure it as a progressive web app that can be easily installed and even planning to add push notifications to let user know of upcoming prayers.</p> <p>Swagger documentation for the API is reachable at <a href="https://prayertimes.api.abdus.dev/docs" target="_blank" rel="noopener">prayertimes.api.abdus.dev/docs</a></p> <h2 id="usage">Usage</h2> <ol> <li>Find a location first by narrowing down country &gt; city &gt; region to get a location id. Or search by country / state / city name:</li> </ol> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">curl <span class="z-s2">&#34;https://prayertimes.api.abdus.dev/api/diyanet/search?q=istanbul&#34;</span></code></pre> <p>You'll get a list of locations:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-err">...</span> <span class="z-p">{</span> <span class="z-nt">&#34;id&#34;</span><span class="z-p">:</span> <span class="z-mi">9541</span><span class="z-p">,</span> <span class="z-nt">&#34;country&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;TÜRKİYE&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;city&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;İSTANBUL&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;region&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;İSTANBUL&#34;</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;id&#34;</span><span class="z-p">:</span> <span class="z-mi">9547</span><span class="z-p">,</span> <span class="z-nt">&#34;country&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;TÜRKİYE&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;city&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;İSTANBUL&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;region&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;ŞİLE&#34;</span> <span class="z-p">}</span> <span class="z-p">]</span></code></pre> <ol start="2"> <li>Using the location <code>id</code> fetch the schedule of prayer times</li> </ol> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">curl <span class="z-s2">&#34;https://prayertimes.api.abdus.dev/api/diyanet/prayertimes?location_id=9541&#34;</span></code></pre> <p>you'll get a json like this:</p> <pre data-lang="json" data-highlighted="true" class="language-json z-code z-chroma"><code class="language-json" data-lang="json"><span class="z-p">[</span> <span class="z-p">{</span> <span class="z-nt">&#34;date&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;2020-01-21T00:00:00&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;fajr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;06:48&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;sun&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;08:17&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;dhuhr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;13:20&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;asr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;15:50&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;maghrib&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;18:13&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;isha&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;19:37&#34;</span> <span class="z-p">},</span> <span class="z-p">{</span> <span class="z-nt">&#34;date&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;2020-01-22T00:00:00&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;fajr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;06:47&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;sun&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;08:17&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;dhuhr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;13:20&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;asr&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;15:51&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;maghrib&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;18:14&#34;</span><span class="z-p">,</span> <span class="z-nt">&#34;isha&#34;</span><span class="z-p">:</span> <span class="z-s2">&#34;19:38&#34;</span> <span class="z-p">},</span> <span class="z-err">...</span> <span class="z-p">]</span></code></pre> Sat, 18 Jan 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/deutscher/ https://abdus.dev/projects/deutscher/ <p>Deutscher is a collection of APIs to fetch example sentences related to a phrase, and perform translations. It uses following sources to find sentences:</p> <ul> <li><a href="https://duden.de" target="_blank" rel="noopener">Duden.de</a></li> <li><a href="https://www.dict.cc/" target="_blank" rel="noopener">Dict.cc</a></li> <li><a href="https://www.deepl.com/" target="_blank" rel="noopener">Linguee/DeepL</a></li> <li><a href="https://context.reverso.net/" target="_blank" rel="noopener">Reverso Context</a></li> </ul> <p>The API is reachable at <a href="https://deutscher.app.abdus.dev/docs" target="_blank" rel="noopener">deutscher.app.abdus.dev/docs</a></p> <h2 id="todo">Todo</h2> <ul> <li>A web app optimized for mobile in front to consume the API.</li> <li>Export to Anki with an option to generate proper HTML markup.</li> </ul> Thu, 16 Jan 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/projects/ascii/ https://abdus.dev/projects/ascii/ <p>This is a simple PWA app for <a href="http://www.figlet.org/" target="_blank" rel="noopener">figlet</a> to generate ASCII word art with hundreds of fonts to choose from. It uses <a href="https://github.com/patorjk/figlet.js" target="_blank" rel="noopener">figlet.js</a> library to render ASCII text.</p> <image-gallery class="image-gallery" role="region" aria-label="Media gallery"> <div class="image-gallery__track" tabindex="0" aria-label="Scroll to see more images and videos"> <img src="./artscii.png" alt="artscii app screenshot" loading="lazy"> </div> </image-gallery> <p>The app is reachable <a href="https://artscii-abdusdev.vercel.app/" target="_blank" rel="noopener">here</a>. The source is available on:</p> <a class="repo-card" href="https://git.abdus.dev/abdus/ascii-pwa" target="_blank" rel="noopener noreferrer"> <span class="repo-card__name"> <span class="repo-card__owner">abdus</span> <span aria-hidden="true">/</span> <strong>ascii-pwa</strong> </span> <span class="repo-card__meta"> <span>JavaScript</span> <span aria-label="0 stars"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-star" /> </svg> 0 </span> <span aria-label="0 forks"> <svg aria-hidden="true" focusable="false" width="1em" height="1em"> <use href="/assets/images/repo-icons.svg?h=817c5427d1ef8a5c1475#repo-fork" /> </svg> 0 </span> </span> </a> Mon, 06 Jan 2020 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/thinkpad-p1-bootloop-fix/ https://abdus.dev/posts/thinkpad-p1-bootloop-fix/ <p>Lately I've been using a Lenovo Thinkpad P1 Gen 2, my first, well actually second, ThinkPad. Today I've depleted the battery completely, then let it hibernate. When I plugged the A/C adapter back and pressed the power button, it didn't turn on, but kept boot-looping.</p> <p>I'm documenting this fix in case someone else encounters the same issue.</p> <h2 id="problem">Problem</h2> <p>Lenovo Thinkpad P1 Gen 2 doesn't turn on. Keyboard lights up then it resets. Force resetting doesn't help.</p> <h2 id="fix">Fix</h2> <ul> <li>Remove the A/C adapter.</li> <li>Locate the emergency reset hole (#1 on the image below). See the <a href="https://pcsupport.lenovo.com/tr/en/products/laptops-and-netbooks/thinkpad-x-series-laptops/thinkpad-x1-extreme-gen-2/manuals/um922835" target="_blank" rel="noopener">manual</a> under <strong>Meet Your Computer &gt; Bottom</strong> heading for reference.<br /> <img src="/posts/thinkpad-p1-bootloop-fix/p1_bottom.jpg" alt="bottom view of Lenovo P1" /></li> <li>Find something pointy like a pin/toothpick/paperclip and stick it in the emergency reset hole under the body.</li> <li>Turn the PC back on.</li> </ul> <p>I guess it was that easy, but I admit that I got flashbacks from trying to fix my old Dell XPS 13. Having a hardware reset button is cool.</p> Tue, 31 Dec 2019 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/youtube-oembed/ https://abdus.dev/posts/youtube-oembed/ <p>Youtube supports <a href="https://oembed.com/" target="_blank" rel="noopener">oEmbed</a> format. This means we can easily fetch information about a video, such as its thumbnail image, HTML for embedding, title etc. without having to use Youtube API or scraping the Youtube page. You can reach oEmbed API at:</p> <pre data-lang="url" data-highlighted="true" class="language-url z-code z-chroma"><code class="language-url" data-lang="url">https://www.youtube.com/oembed?url=YOUTUBE_URL</code></pre> <p>It returns a JSON filled with metadata. You can explore the oEmbed response yourself using the demo below.</p> <h2 id="demo">Demo</h2> <p>Paste in a url and click fetch to inspect an oembed json for a Youtube video.</p> <form id="yt" @submit.prevent="handleSubmit()" x-data="app()" x-cloak> <p> <label for="url">URL:</label> <input id="url" type="url" x-model="url" name="url" placeholder="https://url/to/youtube/video"> </p> <p> <span>oEmbed URL:</span> <pre class='snippet' data-lang="url" x-text="url ? oembedUrl(url) : '...'"></pre> </p> <p> <button type="submit" class="button">fetch!</button> </p> <pre class="snippet" data-lang="json"><code x-text="text"></code></pre> </form> <script src="/assets/js/alpine-3.17.4.min.js" defer></script> <script> function app() { return { url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ', text: '// json will appear here', async handleSubmit() { if (!this.url) return; this.text = 'fetching json...'; // https://www.youtube.com/oembed doesn't send CORS headers, so I have to use a proxy here const url = `https://cors.abdus.dev/?url=${encodeURIComponent(this.oembedUrl(this.url))}`; const json = await fetch(url).then(r => r.json()); this.text = JSON.stringify(json, null, 2); }, oembedUrl(url) { return `https://www.youtube.com/oembed?url=${encodeURIComponent(url)}` } }; } </script> <style> [x-cloak] { display: none; } </style> Sat, 28 Dec 2019 00:00:00 +0000 Abdussamet Kocak https://abdus.dev/posts/ssh-agent/ https://abdus.dev/posts/ssh-agent/ <p>I have several servers that I use to host my projects. Sometimes I need to move data between the servers, but realizing a server is missing an SSH key to another is annoying. This is where SSH agent comes into play. It holds a keychain for SSH keys you choose and forwards them as you ssh between the servers. No more using <code>ssh-copy-id</code> or manually editing <code>~/.ssh/authorized_keys</code>.</p> <p>I'm also fan of <a href="https://fishshell.com/" target="_blank" rel="noopener">fish shell</a>. It's quite handy, but poses a difficulty when using snippets from the internet.</p> <h2 id="usage">Usage</h2> <p>Running <code>ssh-agent</code> sets a few environment variables. <code>ssh-add</code> adds default SSH key to keychain . <code>eval</code> is for running the commands manually in fish shell that would work natively in bash.</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash"><span class="z-nb">eval</span> <span class="z-o">(</span>ssh-agent -c<span class="z-o">)</span> ssh-add</code></pre> <p>running these gives:</p> <pre data-lang="bash" data-highlighted="true" class="language-bash z-code z-chroma"><code class="language-bash" data-lang="bash">abdus@home ~&gt; <span class="z-nb">eval</span> <span class="z-o">(</span>ssh-agent -c<span class="z-o">)</span> Agent pid <span class="z-m">7282</span> abdus@home ~&gt; ssh-add Identity added: /home/abdus/.ssh/id_rsa <span class="z-o">(</span>/home/abdus/.ssh/id_rsa<span class="z-o">)</span></code></pre> <p>After setting up the agent, we can connect to a server from a different server without setting up SSH keys first.</p> <pre data-lang="shell" data-highlighted="true" class="language-shell z-code z-chroma"><code class="language-shell" data-lang="shell">abdus@home ~&gt; <span class="z-nb">eval</span> <span class="z-o">(</span>ssh-agent -c<span class="z-o">)</span> abdus@home ~&gt; ssh-add abdus@home ~&gt; ssh abdus@server1 abdus@server1 ~&gt; ssh abdus@server2 abdus@server2 ~&gt; <span class="z-c1"># it works!</span></code></pre>