[ { "title": "Your First Business Should Be An Excel Sheet", "url": "https://endler.dev/2021/first-business/", "body": "One of the best decisions I made in 2020 was to open my\ncalendar to everyone. People book appointments\nto chat about open-source projects, content creation, and business ideas.\nWhen we talk about business ideas, people usually gravitate towards problems\nsuitable for startups. Things like using artificial intelligence to find clothes\nthat fit or building a crowdfunding platform on the blockchain.\nThese are all great ideas, but they require lots of persistence and deep\npockets. It's probably easier and less risky to just join a startup in that field.\nIn reality, most people just want something cool to work on and happy customers.\nIt turns out, you don't have to run a startup for that. Instead, starting a side\nproject is less risky and can grow organically into a business over time.\nUsually, the solution is right in front of them: in some Excel spreadsheet on\ntheir computer.\n\n \n \n \n \nI Hate Excel\nI never spend more time in Excel than I absolutely have to. Normally, I need to\nget something done quickly. I don't care about the layout or the design; I care\nabout getting over it. Probably I'd pay someone to do that work for\nme β€” and that's my point!\nThe spreadsheets and lists you create for your own solve a real problem: yours;\nand chances are, someone else has the same problem. That's a business\nopportunity!\nThis approach ticks all the boxes:\n\nπŸ’ͺ Solves a real problem.\nπŸ₯± It's boring, so people might pay for not having to do it themselves.\n⚑️ No time wasted on design or infrastructure. It's the ultimate MVP.\n🐒 Low tech: no programming required. You can start with Notion + Super.so.\n🐜 Niche market: if an established service existed, you'd use it.\nBigCorp ain't gonna eat your lunch.\nπŸš€ You spend less time building and more time talking to potential customers.\n\n\n \n \n \n \nExamples\nA few years ago, I looked for static code analysis tools. I did some boring\nresearch, started a list, pushed it to Github, and that's\nthat. Fast forward a few years, and that side\nproject is pulling in money from sponsors and\nattracts consulting gigs.\nAnother example: a guy built a spreadsheet for places to work from remotely. He\nshared the spreadsheet on Twitter. People added more places, and he created a\nwebsite from it. The website is NomadList, and the guy\nis Pieter Levels. He pulls in 300k/year from that website.\n\nInstead of building a site first, I simply made [a] public Google spreadsheet\nto collect the first data and see if there’d be interest for this at all.\nβ€” Pieter Levels on how he created\nNomadList.\n\nI left a spot for your story here. Now brush up that spreadsheet (or that\nlist), share it with your friends, iterate on their feedback, and build your\nfirst business.\n" }, { "title": "Starting A Print-On-Demand Business As A Software Engineer", "url": "https://endler.dev/2021/codeprints/", "body": "One day I had the idea to make a print of my Github timeline.\nI liked the thought of bringing something "virtual" into the real world. πŸ˜„\nSo I called up my friend Wolfgang and we built codeprints.\nIt's my first "physical" product, so I decided to share my learnings.\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Felix Krause of fastlane fame was one\nof our first customers and we are very thankful for this tweet promoting our\nservice, which gave us a huge traffic boost.\n \n \n \n \n \n\n\n \n \n \n \nLaunching Is Hard, So Launch Early\nEven though I knew that launching early was vital, I still didn't want to\n"commit" to the final design shortly before the planned go-live. There was always that last bug to fix or that little extra feature to implement.\nFor example, I wanted to offer two designs/layouts: the classic Github contribution timeline and a graph-based design for repositories.\nIn cases like that, it helps to have a co-founder.\nWolfgang convinced me that multiple layouts were not needed for the MVP and that whatever we'd come up with would probably be wrong anyway without getting early user feedback.\nHe was right. Without Wolfgang, the shop would probably still not be live today.\nWe have a much clearer vision now of what people want to see, thanks to launching early. Turns out users were not really interested in the graph-based design after all, and it would have been a waste of time to create it.\n\n Lesson learned:\nEven if you know all the rules for building products, it's\ndifferent when applying them in practice for the first time. We'll probably\nnever be completely happy with the shop functionality, but it's better to launch\nearly and make incremental improvements later.\n\n\n \n \n \n \nSoftware Development Is Easy\nWhen we started, my main concern was software development. The frontend and the\nbackend needed to be coded and work together. We didn't want to run into Github rate-limiting issues in case there were many users on the site. I was also\nthinking a lot about which web frontend to use. Should we build it in Rust using\nYew or better go with Gatsby?\nTurns out writing the code is the easy part.\nBeing software engineers, it didn't take us too long to implement the backend\nAPI and we quickly found a decent template for the frontend. Most of our time\nwas spent thinking about the product, the user experience,\nfinancing, taxes, the shipping process, marketing, and\nintegrating customer feedback.\nThese were all things I had (and still have) little experience in.\nWolfgang suggested to "just use Shopify and the default template" to get started\nquickly. In hindsight, it was the absolute right decision. I always thought\nShopify was for simple mom-and-pop stores, but it turns out it's highly\ncustomizable, integrates well with pretty much anything, and offers excellent tooling\nlike themekit. Payments, refunds,\ndiscounts, customer analytics: it's all built into the platform. It\nsaved us sooo much development time.\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n\n Lesson learned:\nThere are many unknown\nunknowns\nβ€” things we are neither aware of nor understand β€” when starting a project.\nTry to get to the root of the problem as soon as possible to save time and avoid\nthe sunk cost fallacy.\n\n\n \n \n \n \nUsers Expect Great UI/UX\nGiants like Amazon, Facebook, and Netflix have raised customer\nexpectations for great UX. They spend millions polishing their websites and getting every detail right. As a result, their sites work just right for millions of customers and on every device.\nAn indie shop does not have these resources. Nevertheless, many customers expect the same quality user experience as on other sites they use.\nBeing on the other side of the fence for the first time, I learned how hard it\nis to build a user interface that works for 90% of the people. Every little\ndetail β€” like the order of form fields β€” makes a huge difference. Get too\nmany details wrong, and you lose a customer.\nThose things can only be found by watching real users use your product. I promise you, it will be eye-opening!\n\n Lesson learned:\nWatch potential customers use your service. It will be\npainful at first, but will improve the quality of your product. Use standard\nframeworks for shops if you can because they get many UI/UX details\nright out of the box. WooCommerce or\nShopify come to mind.\n\n\n \n \n \n \nBuilding Products Means Being Pragmatic\nWe have many ideas for future products. Many friends and customers tell us about\npotential features all the time, but the problem is how to prioritize them.\nMost ideas won't work at scale: It's tricky to find a supplier that has a\nproduct on offer, is cheap, ships worldwide, and has a working integration with\nyour shop-system. So we have to regularly scrap product ideas, simply\nbecause our suppliers' support is not there. On top of that, we run the\nbusiness next to our day job and other\nresponsibilities, so we need to make use of our time as efficiently as possible.\n\n Lesson learned:\nMaking services look effortless is hard work. Time is your biggest constraint.\nYou'll have to say "no" more often than you can say "yes".\n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Due to the pandemic, codeprints was\nentirely built remotely. More people should give whereby\na try.\n \n \n \n \n \n\n\n \n \n \n \nGetting Traction As A Small Business\nIt has never been easier to launch a shop. Services like Shopify, Stripe, and a\nhost of suppliers make starting out a breeze. On the other hand, there is a lot\nmore competition now that the barrier to entry is so low.\nThousands of services are constantly competing for our attention. On top of\nthat, most customers just default to big platforms like Amazon, AliExpress, or eBay\nfor their shopping needs these days, and search engines send a big chunk of the traffic there.\nSince our product is custom-made, we can not offer it on those bigger platforms.\nAs an indie shop, we get most visitors through word of mouth, exceptional\ncustomer support, and advertising where developers hang out:\nTwitter, Reddit, HackerNews, Lobste.rs, and friends. It's essential to focus on\nproviding value on those platforms; a plain marketing post won't get you any attention. Other\nplatforms like LinkedIn, Facebook, ProductHunt, or IndieHackers could also work, but our target audience (OSS developers with an active Github profile) doesn't\nhang out there that much.\n\n Lesson learned: Always know where your customers are and understand their needs.\n\n\n \n \n \n \nFinding A Niche Is Only Half The Job\nCommon market wisdom is to find niche and grow from within. With codeprints we definitely found our niche: the audience is very\nnarrow but interested in our geeky products. There are 56 million developers on\nGithub today; that's a big target audience. Most profiles are not very active,\nthough. To make a print look attractive, you'd have to consistently commit\ncode over a long period of time β€” many years. If we assume that only 1% of\ndevs are active, that limits our target audience to 560.000 users. That's still\na big but much smaller market. Now, if only 1% of these people find the shop and\norder something (which would be quite a good ratio), we're looking at 5.600\norders total. Not that much!\nIn order to extend that audience, one could either increase the number of\npotential customers or focus on getting more of the existing potential customers\non the page.\nIn our case, we expanded by offering a one-year layout, reducing the\nrequired level of Github activity for a cool print. We are also working on making\nemptier profiles look more interesting and highlighting the value-producing part\nof open source contribution. Every contribution counts β€” no matter how tiny.\n\n Lesson learned:\nMake sure that your niche market is not too narrow so that you can make a sustainable business out of it.\n\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Early adopters like Orta\nTherox are incredibly precious when starting out. Not\neverybody has a rockstar profile like that, though (and that's fine).\n \n \n \n \n \n\n\n \n \n \n \nMake User Feedback Actionable\nInitial customer feedback is precious. You should focus on every word these\ncustomers say as they believe in your product and want you to win. (They voted\nwith their wallet after all.) Feedback from\nfriends is helpful, too, but I usually apply a bigger filter to that. Not all\nof my friends are software developers, and while they all mean well, what they\ntell me might be different from what they mean. It's like they\nare asking for faster\nhorses when what they\nreally want is a car.\nFeedback on social media can be... snarky at times; be prepared for that! Your job\nis to find the grain of truth in every statement and focus on constructive\nadvice.\nFor example, take this feedback we got:\n\nHow lazy can someone be to pay €36 for this.\n\nYou could turn it around to make it constructive:\n\nCan I get a cheaper version to print myself?\n\nAnd that is some valuable feedback. We could provide a downloadable version in\nthe future!\n\n Lesson learned:\nIt takes practice to extract actionable feedback from user input and make it fit your product vision.\n\n\n \n \n \n \nSummary\n2020 was a crazy year.\nI helped launch two small side-businesses, codeprints and\nanalysis-tools.dev.\nBoth have an entirely different revenue model, but\nthey have one thing in common: they were super fun to build! 🀩\nIt's motivating to look back at those achievements sometimes...\nThat print of 2020 pretty much encapsulates those feelings for me.\n(Note the greener spots in August and September, which is when we launched\nanalysis-tools and the days in December when we built codeprints.)\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n My coding year in review using our new\nvertical layout.Here's to\nbuilding more products in 2021.\n \n \n \n \n \n\nLet me know if you found that post helpful and reach out if you have questions.\nOh and if you're looking for a unique way to decorate your home office, why not\nget your own print from codeprints? 😊\nP.S.: If you're a product owner and you're looking for a unique present for your\nteam, get in contact and be the first to get an invite to a private beta.\n" }, { "title": "So You Want To Earn Money With Open Source", "url": "https://endler.dev/2021/oss-money/", "body": "I earned 0 Euros from maintaining OSS software for years, and I thought that's\nthe way things are. I finally looked into ways to monetize my projects last year\nand in this talk I want to share what I learned so far. It didn't make me\nrich (yet!), but I built my first sustainable side-project with\nanalysis-tools.dev ✨.\nI'll talk about this and other projects and the mistakes I made on the road\ntowards sustainability.\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\nRelated links and resources:\n\nPodcast by Caleb Porzio about why building\na business around Github sponsors is so\nhard.\nThe Changelog Podcast β€” It’s OK to make money from your open source\nwith Zeno Rocha.\nNadia Eghbal talking about Maintenance of our essential info-structure.\n\nFind a full transcript of the talk below. (Sorry for the wall of text.)\n\nThis is my talk about earning money with Open Source, which I gave at the Web\nEngineering Meetup Aachen at the end of 2020. The organizers gladly allowed me\nto share it on my YouTube channel. I'm basically trying to answer the question:\n"Why am I not making 100k on Github?". I'm talking about finding corporate\nsponsors for myself and the long road towards sustainability of open-source\nmaintenance.\nYou might not even want to start. This is a talk for those people that have\nthe mindset that it's probably not worth it to spend that much effort on Open\nSource if it takes so long until you find success. Now, this talk turned out to\nbe a little grim. I had this very motivational talk in mind, but in reality,\nit's hard, and by hard, I mean it's really hard.\nI just want to get this point across and maybe still motivate you to do it but\nfirst: why am I entitled to talk about this? I've been doing Open Source for 10\nyears over 10 years now. This is a talk dedicated to my former self maybe 15\nyears ago. I work at trivago, which is a hotel search company based in\nDΓΌsseldorf. I have a blog at endler.dev. Like everyone and their mom, I also\nhave a YouTube channel. It's called Hello, Rust! and I'm\nextremely active with one video every two years. Hence, you definitely want to\nsubscribe to not miss any updates. But today, I want to talk about Open Source,\nand I have a very sophisticated outline with two points my journey and revenue\nmodels.\nLet's go back all the way to 2010. The world definitely looked a bit different\nback then.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Github in 2010\n \n \n \n \n \n\nThis was Github, and I was a bit late to the game. I joined in January 2010, and\nby then, Github was already two years old, so my username was taken. I usually\ngo by the handle mre on platforms, and I noticed that this handle was not\nused by anyone, so I just sent a mail to support and asked if I could have it,\nand then I got an answer from this guy saying "go for it." It was Chris\nWanstrath, who goes by the handle defunct, and he's the former CEO of Github,\nand at this point in time, I was hooked. I really liked the platform. I really\nliked how they worked very hands-on with Open Source. I used it for some\nprojects of mine; you can see in the screenshot that I uploaded my blog, for\nexample, because they host it for free. It was built with Jekyll, and you just\npush it to their site. Then they statically generate it, and it's done. It goes\nwithout saying that nothing has changed in the last 10 years because my blog\nmore or less still looks like that. It's not built with jQuery and Jekyll\nanymore, but with zola and Cloudflare Worker Sites, but it's more or less the\nsame thing. For preparing for this talk, I wanted to take a step back and see\nwhere I was coming from and where I am right now, and probably the best way to\ndo it is to look up some statistics and see if the number of repositories over\ntime would give me some insights. So I queried the Github API for that.\nYou can see it's pretty much a linear graph from 2010 all the way to 2020.\nExcept for 2018, where I reached peak productivity, it seems, but oh well. In\nthe end, it's more or less a linear thing, and you might say you put some work\nin you get some feedback out, but in reality, it's different. There is a\ncompound effect. If we look at my number of stars over time, you can see that\nmore or less it started very slowly, and now it's sort of growing exponentially,\nso right now, we are at 25.000 stars across all projects. Another way to look at\nit would be the number of followers. That's kind of a new metric to me, but I\ndid look up some statistics from archive.org (because Github doesn't have that\ninformation through their API), and again, it's more or less exponential growth.\nYou put some work in, but you get a compound effect of your work plus some\ninterest out. This is not luck; it's work. It means you know what you're doing.\nAt the same time, there's the elephant in the room, and that is it's just a pat\non the back. We have earned zero dollars until now, and one question you might\nhave is how do you monetize this effort.\nFirst off, is it an effort?\nWell, I don't know about you, but I probably spend two or three hours on average\nper day on Open Source: thinking about Open Source and creating new projects,\nbut also maintaining and code review, so it really is work, and it's a lot of\nwork, and you more or less do that for free.\nThere's nothing wrong with doing things for free and doing it as a hobby, but in\nthis case, you are supposed to be working on whatever you like. Open Source is\nnot like that; sometimes you have obligations, and you feel responsible for\nmaybe helping people out, which is a big part of it. You do that next to your\nregular work, so it can really be a burden. If you don't know by now, making\nthis somehow valuable is hard, it's really hard. I want to talk about some ways\nto build a proper revenue model from Open Source. It goes without saying that\nthis should probably not be your first focus if you saw the graphs before, but\nonce you reach a point where you want to get some revenue, you have a couple of\noptions. I don't want to talk about doing Open Source as part of your business,\nand I don't want to talk about bigger companies and more significant support\nhere. I want to focus on a couple things that everyone can do. Sponsoring [on\nGithub] is one. Offer paid learning materials on top of your normal\ndocumentation. For example, you might have a video series that you ask for\nmoney. Sell merchandising like Mozilla does. Consulting next to your Open Source\nbusiness Services and plugins like writing an ADFS plugin or high availability\nfunctionality are very common examples for paid features targeting enterprises.\nBut let's start with the basics. Let's start with point number one, sponsoring.\nThere are two types of sponsoring: the first one is individual donations.\nIndividual sponsoring is what Github Sponsors is all about. If you want to earn\nmoney [with that model], you have to think about the funnel, and you have to\nthink about how you capture people's attention and how you monetize that. It\nstarts with a product, [which] can be anything. From there, you generate\ninterest, and this interest creates an audience, and that audience eventually\nmight pay for your service, and this is actually the entire secret. It's how you\nearn money with any product, and with Open Source, if you want to attract\nsponsors, you build a product people want.\nIf you transfer that to Open Source, building a project is maybe a repository,\nand the stars indicate the interest of the audience. The audience itself is made\nout of followers (personal followers or followers of a company), and those\nfollowers might or might not become sponsors in the end. Now, I know stars are a\nterrible metric for popularity because some people use stars differently than\nothers. For example, some use it as bookmarks to check out projects later,\nothers want to thank the developers for maybe putting in a lot of effort, and so\non, but it's a good first estimation.\nNow, think about the following. Think about the number of stars I have and the\nfollowers and the number of sponsors. Think about my "funnel" right now. I told\nyou that I have 25.000 stars and roughly 1000 followers, and out of those, I\nhave three sponsors, so the ratio between the stars and sponsors is 0.01. That\nlooks pretty grim. It means you need around 8.000 stars to attract a single\nsupporter. I was wondering: "maybe it's just me?". Maybe the top 1000 Github\nmaintainers did not have that problem. Well, it turns out it's exactly the same\nschema. If you take the top 1000 Github maintainers and look at their sponsors,\nit's again a pretty grim picture. For example, looking at the median, you look\nat 3421 followers per person and a median of zero sponsors. That's zero percent\nif my math is correct, and if you look at the average, you even have 5430\nfollowers (because Linus Torvalds pushes that number up). You have 2.8 sponsors\nout of that on average, and that is 0.5%, which is a bit more than I have, but\nit's roughly in the same ballpark. Now think about this: Github has 40 million\nusers, so that means the top 1000 maintainers make up 0.0025% of the entire\ncommunity. The median income of those maintainers on Github is basically zero.\nThat in and on itself is maybe not the biggest problem, but keep in mind that\nthe Github revenue of 2019 was 300 million dollars. I read that comment on\nHacker News yesterday:\n\nI have sponsors on Github and rake in a cool two dollars per month. It's\nobviously less after taxes, so I have to have a day job.\n\nSo this is clearly not working. You have to think of different ways to monetize\nOpen Source, or you just wait until Github Sponsors becomes more popular --\nwhatever happens first. One way I just want to quickly touch on is the notion of\nsponsorware. It's kind of a new concept, and some people haven't heard of it\nbefore. I honestly really like it. Generally speaking, you create a project, and\nyou keep it private. You talk about it on Twitter, though or any other platform,\nand you say: "hey, I'm building this, but if you want early access, you have to\nbecome a sponsor," and once you reach a certain threshold of sponsored sponsors,\nor income or whatever. Then you make a project public. This initial example that\nI showed you, where someone was earning 100k on Open Source, is from someone\ndoing just that. He's building products and services, talks about them, and then\nmakes them open for everyone in the end.\nThis has some advantages: first of you get early feedback from people that\nreally believe in your mission. Second, you don't have to work for free all the\ntime, and third, you might also create an audience and hype from your projects.\nThe disadvantage is that if you are a hardcore Open Source or free software\nbeliever, this goes against your ethic. You want the software to be open, to\nbegin with, without any additional requirements. So you really have to make up\nyour own mind about that. I tried, and I have an early access program, which I\nonly share with sponsors. [My first sponsorware was a] tool for getting Github\nstatistics. [The statistics from this talk were] created with that tool. I think\nyou need a big audience to pull that off. The question is if you want to put\nthat much effort in, or you just want to make it open in the first place and\nthink about other revenue models. However, I think still it's a very interesting\nconcept, and we might see that [more] in the future, so you know how it looks\nlike now, and you have a name for it.\nAnother one is corporate sponsoring. This is a double-edged sword because\ncorporate sponsoring means that a company gives you money and sometimes wants\nsomething. They might want additional support, or they want the bug to be fixed,\nand more or less it feels like you are somehow beginning to work for them, but\nnevertheless, those companies put in quite a big amount of money into Open\nSource these days. Looking at two big companies, Facebook and Google, they\ninvested 177k and 845k respectively into Open Source over their lifetime on Open\nCollective, a platform for collecting those donations. That's really great. We\nneed more companies doing that, but also, as a little side note and maybe as a\nlittle rant, I believe that those companies are doing way too little.\nFacebook's revenue last year was 70 billion, and Google had 160 billion, which\nis nothing to be ashamed of, so I wonder really if this is the most they can do.\nOf course, Google, for example, also donated to other projects like Mozilla, and\nthey also organize meetups and so on. But do you really think that Facebook and\nGoogle would exist today if there was no Python or web server or Linux back in\nthe day when two Stanford students tried to build a search engine? Sometimes I\nfeel that Fortune 500 companies really don't understand how much they depend on\nOpen Source and how many people depend on a few people who maintain critical\nparts of our infrastructure.\nI don't think they invest nearly enough into Open Source. What a lot of people\nthink is that Open Source works like the panel on the left where you have a full\nroom of engineers trying to figure out the best way to build a project, and in\nreality, it's more or less someone working late at night to fix bugs and doing\nit because they believe in it. The public perception is probably wrong, and a\nreally small group of people who maintain critical infrastructure. Sometimes\nthat can lead to very tricky situations. Two of my childhood heroes talked about\nit openly: Kenneth Reitz is the core maintainer of\nrequests for Python and antirez is the creator of\nRedis, a key-value store. So one is front-end development\nand the other one from backend-end. They both talk about burnout here because\nthe burden of becoming an Open Source maintainer on a big scale can very much\nand very quickly lead to burnout. The internet never sleeps. You never go to\nsleep. You always get a ticket, a feature request, a pull request, an issue. You\nalways have something to work on, and on top of that, you have to do all your\nother responsibilities, so that can lead to burnout really quickly. There was\none guy who I also respect deeply. His name is Mark Pilgrim. He is the author of\nDive Into Python, and he once pulled a 410 for deleting everything [about him]\non the internet. There's actually a term for it: infocide for "information\nsuicide." He got fed up with the ecosystem, and if you think about the Ruby\ncommunity, you might remember _why, the author of the Poignant Guide to Ruby.\nHe did kind of the same thing. Focusing on what antirez has said, "once I\nstarted to receive money to work at Redis, it was no longer possible for my\nethics to have my past pattern, so I started to force myself to work on the\nnormal schedules. This, for me, is a huge struggle for many years. At this\npoint, moreover, I'm sure I'm doing less than I could, because of that, but this\nis how things work", so it feels like he feels guilty for maybe being forced\ninto that work schedule and maybe not performing well enough. There are some\nsigns of burnout for me somehow, and it's that love-hate relationship of Open\nSource and money. If you accept money, it becomes a job, but you're not writing\ncode most of the time. You're writing the talks, reviewing pull requests, you're\nlooking at issues, you're answering questions on StackOverflow, you're\ndiscussing on Discord, you're marketing on YouTube or conferences. When you\nbecome popular with Open Source, then it feels like you have a choice between\ntwo options: one is depression and the other one is burnout. If your project\ndoes not become successful, then suddenly you think you're a failure, you're a\nmistake. It has zero stars; nobody likes it. But if it becomes a success, then\neveryone likes it, and you get hugged to death. That's a really unfortunate\nsituation to be in, and you want to stop being overwhelmed with those\nresponsibilities. You have to set clear boundaries and pick your poison. You\nhave to be careful if you accept companies as sponsors. I want to show you one\nexample of how it can work and [point out] some risks. Earlier this year, I\nstarted working on a real project that I had been putting off for many years\nbefore.\nYou see, in December 2015, I started a list of static analysis tools on Github.\nStatic analysis tools are just tools that help you improve your code, and it\nturns out that there's a lot of those tools. Just starting to collect them was\nthe first step. I didn't think much about it, but over time that became really\npopular. And you can see that this graph is more or less a linear increase in\nstars over time. In 2018, I started really thinking hard about whether there was\nmore than just a Github project here. I talked to many people that I had this\nidea of building something more from that. It really took someone else to maybe\npush me over the finishing line and convinced me that this was worth it, and\nthat is Jakub. He said, "why not build a website from it?" and over the course\nof maybe two weekends or so, we built a website. It's built with Gatsby, but it\nreally doesn't matter. We just did it, and then we saw what happened to it. We\nrender 500 tools right now, and the initial feedback was really great. People\nreally seem to like that. We got a cool 720.000 requests on the first day, and\nover the next week or so, it more or less hit 1.5 million. That was great\nbecause suddenly people started getting interested in that project. So we\nstarted finding some sponsors. Those companies are special because they believe\nin your mission, but they also know how Open Source works. They don't really\nexpect you to advertise their tool. They want to sell to developers, so they\nwant to be in the developers' minds, saying: "Hey! You are a developer. We built\nthis amazing tool you might want to check it out!" but they also get seen as an\nOpen Source company. I think that's a win-win. I have to say it doesn't always\ngo as easily. sometimes companies expect you to just have cheap advertising\nspace. Then they jump off the moment they see you don't get that many clicks,\nbut others understand that they invest into something that maybe pays off in a\nyear or two from now. So I'm really thankful that some companies understand that\nmission. However, what companies want is different than what individuals want.\nCompanies want an invoice. Companies want something tax-deductible. Companies\nwant someone that keeps the lights on and is responsive via email, so you really\nhave those obligations, and one platform that helps with that is Open\nCollective. They have a 501c6 program for Open Source projects that acts as a\nfiscal host, which means they will do all the invoicing and officially be the\nmaintainers. If you, as an Open Source maintainer or a contributor to a project,\nwant to get [reimbursed for your work], you have to send an invoice to open\ncollective.\nI think that's the best of both worlds. Again, because it's a very transparent\nprocess, companies are in the loop and don't have to deal with all the financial\nstuff. But it also means that you have to really polish your public perception.\nCompanies really want to know what they can get out of sponsoring you, and you\nhave to make that very clear. Probably the most important site that you have is\nnot your website, but it's your sponsors page on Github where you describe the\ndifferent tiers and what those tiers mean, so we have three tiers: One is\ntargeted at smaller companies and freelancers. They just get exposure, and they\nget seen as an Open Source friendly tech company. That is a hundred dollars a\nmonth. We have a middle-tier, a company sponsor that maybe is a bigger company.\nThey get the batch, too, but they also get a blog post about a static analysis\ntool that they want to promote, but we make it transparent that this is really a\nsponsored content. Finally, if you want to go all the way, you go to full\ncontent creation, which might be a video workshop, but we don't have video\nworkshop sponsors yet, so I cannot talk about that yet. I have to say I really\nwould like to try though and it's cheap really for what you get.\nAnyway, those are things that you can do today. Without really changing how you\nwork on Open Source, you can set that up, and you just see how it goes. Maybe no\none reacts, and that's fine. Everything else on that list is kind of advanced.\nYou need an audience, and so you should start with that.\nPaid learning material is something that we are doing with analysis tools in the\nfuture with a video course. There are companies like tailwind that do that\nimpressively well, so you can learn from them. For merchandising, you have to\nhave a brand. Hence, it's not something that I could do, but someone like\nMozilla or the Coding Train on\nYouTube could definitely do something like that. Consulting is always an option.\nStill, it's also a lot more work and probably takes you away from what you\nreally love, so it really becomes a job. You have to think about whether you\nwant to do that or not. Enterprise services are very advanced [and interesting]\nfor maybe the one percent of projects that can be run in a business and where\nyou have special requirements. I have to say start from the top and work your\nway down. Start to create an audience. It's probably easier to build an audience\non Twitter and then funnel it back to Github than the other way around. Oh, by\nthe way, did I tell you it's hard? I really don't want to end on a low note. I\nreally want to emphasize that I would do it again, all of that if I started\ntoday. I think there's no better time to contribute to Open Source than today.\nProbably tomorrow will even be a better time because suddenly, way more people\nare interested, it's way easier to set up projects, you have all those free\ntools like VSCode and Github actions, free hosting. It's just amazing how much\nyou can pull off with very little money involved. So you can try it. What's the\nworst thing that can happen? No one cares? Well, okay, then you're as good as\nme. But I have some tips for you if you want to start today. My first tip is:\n"do your homework." Many people start with learning, and then they build things,\nand then they close the circle, but there's one key piece missing here. Some\npeople hate the word, but you learn to love it eventually. It's called\nmarketing. Marketing means a lot of things to a lot of people, but what it means\nto me is getting the word out because someone else will if you don't, and you\nare awesome; you just have to realize that. Maybe not everyone knows [about your\nproject] right away, so you should really talk about it more. Maybe at\nconferences, maybe on Twitter, maybe you can just tell your friends. Maybe you\ncan ask people to contribute and to support you. Somehow it's frowned upon in\nthe community that if you do marketing, you're not doing it for real, but I\nthink that's not true. I think that if smart people and patient and passionate\npeople did marketing, then the world would be a better place; because I'm pretty\nsure the evil guys do marketing. So do your homework, but rest assured that\nbeing an Open Source maintainer means running a business, and you are the\nproduct. You have to think about why someone would want to sponsor you because\nif you don't come up with an answer for that, how should they know. Also, think\nabout the funnel. How will people find you, for example? The best way for people\nto find you is probably starting a YouTube channel.\nThere are easier ways, though.\n[First,] you can always help out in a different project, and you\ndon't even have to be a coder. If you are good with design, then I can tell you\nthere are so many Open Source projects that need designers. It's crazy.\nMaybe start creating a logo for a small project and start getting some\nvisibility. Another one is having fun. If you know that earning money is hard in\nOpen Source, then that can also be liberating because it means you can\nexperiment and you can be creative, and yeah, having fun is the most important\nthing, I guess.\nSecond, build things you love because it's your free time in the end. The\nchances that someone will find the project is pretty low, so it better be\nsomething that you're really interested in. If you don't believe in that, just\nmove on to the next thing. It's fine if you drop a project that you don't\nbelieve in anymore. No one will hold you accountable for that unless they are\njerks, and you don't want to be surrounded by jerks.\nThird, find friendly people because you really grow with your community. You\nwant people that support your project and maybe eventually become maintainers to\nease the burden, and that takes a lot of time, sometimes years, until you find\none maintainer, so always be friendly, try to put yourself in their perspective.\nGo the extra mile if you can. For example, reintegrate the master branch into\ntheir pull request. Just do it for them. Say thanks twice if you're unsure.\nFourth is to grow an audience.\nRadical marketing is one way, but being approachable and being inclusive is\nanother way. You want to be the guy or the girl that people go to when they have\na tough question, or they want to know how to get into Open Source. You want to\nbe the person that helps them out on their first pull request. They will pay it\nback a thousand times. The most exciting people I have met so far are available\nfor questions, and they don't really ask for anything in return. You hold them\nvery close and dear to your heart. When the time comes, you will remember those\npeople. We will say, like, "this is an amazing person to work with; I can highly\nrecommend them," which is called a lead.\nFinally, be in it for the long run. Good things take time. You see, it took me\n10 years. Maybe it takes you five or maybe even less, but it's probably not an\novernight success. It's really a long-term investment.\n" }, { "title": "My Blog Just Got Faster: Cloudflare Workers and AVIF Support", "url": "https://endler.dev/2020/perf/", "body": "\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n\nDid I mention that this website is fast?\nOh yeah, I did, multiple times.\nFew reasons (from ordinary to the first signs of creeping insanity):\n\nπŸ“„ Static site\n☁️ Cached on Cloudflare CDN\nπŸ”— ️HTTP/2 and HTTP/3 support\n🚫 No web fonts (sadly)\nβœ… Edge-worker powered analytics (no Google Analytics)\n🌸 Avoiding JavaScript whenever possible; CSS covers 90% of my use-cases.\nπŸ–ΌοΈ Image width and height specified in HTML to avoid page reflows.\nπŸ‘πŸ» Inlined, optimized SVG graphics and hand-rolled CSS\nπŸš… Static WASM search (lazy loaded)\n🏎️ The entire homepage is <10K (brotli-compressed), including graphics, thus should fit into the first HTTP round-trip.\nπŸ’Ÿ Heck, even the favicon is optimized for size.\n\nThen again, it's 2020: everyone is optimizing their favicons, right? ...right!?\nWell, it turns out most other sites don't think about their user's data plans as much as I do. Actually, that's an understatement: they don't care at all. But to me, lean is beautiful!\n\n \n \n \n \nWait, What About Images?\nI prefer SVG for diagrams and illustrations.\nOnly if it's a photo, I'll use JPEG or WebP.\nTo be honest with you, I never really liked WebP.\nThe gist is that it might not even be smaller than JPEGs compressed with MozJPEG.\nThere is a lengthy debate on the Mozilla bug tracker if you want to read more.\nTo this day, Safari doesn't support WebP.\n\n \n \n \n \nHello AVIF πŸ‘‹\nMeet AVIF, the new next-gen image compression format. Check this out:\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n Source: ReachLightSpeed.com\n \n \n \n\nIt's already supported by Chrome 85 and Firefox 80.\nThen it hit me like a hurricane πŸŒͺ️:\n\n 😲 Holy smokes, AVIF is supported by major browsers now!?\nI want this for my blog!\n\nYes and no.\nI'm using Zola for my blog, and\nAVIF support for Zola is not yet there, but I want it now!\nSo I whipped up an ugly Rust script (as you do) that creates AVIF images from my old JPEG and PNG images. I keep the original raw files around just in case.\nUnder the hood, it calls cavif by Kornel LesiΕ„ski.\n\n \n \n \n \nData Savings\nThe results of AVIF on the blog were nothing short of impressive:\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n Total image size for endler.dev/2020/sponsors\n \n \n \n \n \n\n\n \n \n \n \nCheck Your Browser\nBut hold on for a sec... is your browser even capable of showing AVIF?\n\n \n \n\nIf that reads "yup," you're all set.\nIf that reads "nope," then you have a few options:\n\nOn Firefox: Open about:config from the address bar and search for avif.\nOn Chrome: Make sure to update to the latest version.\nOn Safari: I'm not sure what you're doing with your life. Try a real browser instead. 😏\n\n\n \n \n \n \nWorkaround I: Fallback For Older Browsers\nHTML is great in that your browser ignores unknown new syntax.\nSo I can use the <picture> element to serve the right format to you. (Look ma, no JavaScript!)\n\n<picture>\n <source srcset="fancy_browser.avif" />\n <source srcset="decent_browser.webp" />\n <img src="meh_browser.jpg" />\n</picture>\n\nThe real\nthing\nis a bit more convoluted, but you get the idea.\n\n \n \n \n \nWorkaround II: Wrong Content-Type On Github Pages\nThere was one ugly problem with Github and AVIF, though: Their server returned a\nContent-Type: application/octet-stream header.\nThis meant that the images did not load on Firefox.\nThere is no way to fix that on my side as Github is hosting my page. Until now!\nI wanted to try Cloudflare's Workers Sites for a long time, and this bug\nfinally made me switch. Basically, I run the full website as an edge worker\nright on the CDN; no own web server is needed. What's great about it is that\nthe site is fast everywhere now β€” even in remote locations β€” no more\nroundtrips to a server.\nBy running an edge worker, I also gained full control over the request- and response objects.\nI added this gem of a snippet to intercept the worker response:\n\nif (/\\.avif$/.test(url)) {\n response.headers.set("Content-Type", "image/avif");\n response.headers.set("Content-Disposition", "inline");\n}\n\nAnd bam, Bob's your uncle. Firefox is happy.\nYou can read more about modifying response objects here.\nAnother side-effect of Workers Sites is that a production deployment takes one minute now.\n\n \n \n \n \nPerformance Results After Moving To Cloudflare\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Website response time before\n \n \n \n \n \n Source: KeyCDN\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Website response time after\n \n \n \n \n \n Source: KeyCDN\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Page size and rating before\n \n \n \n \n \n Source: Pingdom.com\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Page size and rating after\n \n \n \n \n \n Source: Pingdom.com\n \n \n \n\nI don't have to hide from a comparison with well-known sites either:\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Comparison with some other blogs I read\n \n \n \n \n \n Source: Speedcurve\n \n \n \n\n\n \n \n \n \nFurther reading\n\nHow to Use AVIF: The New Next-Gen Image Compression Format β€” Nice introduction that highlights some common pitfalls when integrating AVIF. It inspired me to add AVIF support.\nAVIF has landed by Jake Archibald β€” Compares image sizes and qualities of different formats: SVG, JPEG, PNG, WebP, and AVIF.\navif.io β€” Fast, configurable, client-side image compression that works on desktop and mobile.\nSquoosh β€” another image compression service built with WebAssembly that supports AVIF\nTons of great examples on how to configure Cloudflare workers\nCloudflare Workers Sites\n\n" }, { "title": "Launching a Side Project Backed by Github Sponsors", "url": "https://endler.dev/2020/sponsors/", "body": "Yesterday we launched analysis-tools.dev, and boy had I underestimated the response.\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\nIt's a side project about comparing static code analysis tools.\nStatic analysis helps improve code quality by detecting bugs in source code\nwithout even running it.\nWhat's best about the project is that it's completely open-source. We wanted to\nbuild a product that wouldn't depend on showing ads or tracking users. Instead,\nwe were asking for sponsors on Github β€” that's it. We learned a lot in the\nprocess, and if you like to do the same, keep reading!\n\n \n \n \n \nFirst, Some Stats\nEveryone likes business metrics. Here are some of ours:\n\nThe project started as an awesome list on Github in December\n2015.\nWe're currently listing 470 static analysis tools.\nTraffic grew continuously. Counting 7.5k stars and over 190 contributors at\nthe moment.\n500-1000 unique users per week.\nI had the idea to build a website for years now, but my coworker Jakub\njoined in May 2020 to finally make it a reality.\n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Github stars over time. That graph screams BUSINESS OPPORTUNITY.\n \n \n \n \n \n Source: star-history.t9t.io\n \n \n \n\n"Why did it take five years to build a website!?", I hear you ask. Because I\nthought the idea was so obvious that others must have tried before and failed.\nI put it off, even though nobody stepped in to fill this niche.\nI put it off, even though I kept the list up-to-date for five years, just to\nlearn about the tools out there.\nYou get the gist: don't put things off for too long. When ideas sound obvious, it's probably because they are.\n\n \n \n \n \nRevenue Model\nIt took a while to figure out how to support the project financially. We knew\nwhat we didn't want: an SEO landfill backed by AdWords. Neither did we want to\n"sell user data" to trackers.\nWe owe it to the contributors on Github to keep all data free for everyone.\nHow could we still build a service around it?\nInitially, we thought about swallowing the infrastructure costs\nourselves, but we'd have no incentive to maintain the site or extend it with new\nfeatures.\nGithub Sponsors was still quite new at that time. Yet, as soon as we realized\nthat it was an option, it suddenly clicked: Companies that are not afraid of a\ncomparison with the competition have an incentive to support an open platform\nthat facilitates that. Furthermore, we could avoid bias and\nbuild a product that makes comparing objective and accessible.\nSponsoring could be the antidote to soulless growth and instead allow us to build\na lean, sustainable side business. We don't expect analysis-tools.dev ever to be\na full-time job. The market might be too small for that β€” and that's fine.\n\n \n \n \n \nTech\nOnce we had a revenue model, we could focus on the tech. We're both engineers,\nwhich helps with iterating quickly.\nInitially, I wanted to build something fancy with\nYew. It's a Rust/Webassembly framework and\nyour boy likes Rust/Webassembly...\nI'm glad Jakub suggested something else: Gatsby. Now, let me be honest with\nyou: I couldn't care less about Gatsby. And that's what I said to Jakub: "I\ncouldn't care less about Gatsby." But that's precisely the point: not being\nemotionally attached to something makes us focus on the job and not the tool.\nWe get more stuff done!\nFrom there on, it was pretty much easy going: we used a starter template, Jakub\nshowed me how the GraphQL integration worked, and we\neven got to use some Rust! The site runs on Cloudflare as an edge\nworker built on top of Rust. (Yeah, I cheated\na bit.)\nCount to three, MVP!\n\n \n \n \n \nFinding Sponsors\nSo we had our prototype but zero sponsors so far. What started now was (and\nstill is) by far the hardest part: convincing people to support us.\nWe were smart enough not to send cold e-mails because most companies ignore\nthem. Instead, we turned to our network and realized that developers reached out\nbefore to add their company's projects to the old static analysis list on\nGithub.\nThese were the people we contacted first. We tried to keep the messages short\nand personal.\nWhat worked best was a medium-sized e-mail with some context and a reminder that\nthey contributed to the project before. We included a link to our sponsors\npage.\nBusinesses want reliable partners and a reasonable value proposal,\nso a prerequisite is that the sponsor page has to be meticulously polished.\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Our Github Sponsors page\n \n \n \n \n \n\nJust like Star Wars Episode IX, we received mixed reviews: many people never\nreplied, others passed the message\non to their managers, which in turn never replied, while others again had no\ninterest in sponsoring open-source projects in general. That's all fair game:\npeople are busy, and sponsorware is quite a new concept.\n\nA little rant: I'm of the opinion that tech businesses don't nearly sponsor\nenough compared to all the value they get from Open Source. Would your company\nexist if there hadn't been a free operating system like Linux or a web server\nlike Nginx or Apache when it was founded?\n\nThere was, however, a rare breed of respondents, which expressed interest but\nneeded some guidance. For many, it is the first step towards sponsoring any\ndeveloper through Github Sponsors / OpenCollective.\nIt helped that we use OpenCollective as our fiscal host, which handles invoicing\nand donation transfers. Their docs\nhelped us a lot when getting started.\nThe task of finding sponsors is never\ndone, but it was very reassuring\nto hear from DeepCode - an AI-based semantic\nanalysis service, that they were willing to take a chance on us.\nThanks to them, we could push product over the finishing line. Because of them,\nwe can keep the site free for everybody. It also means the website is kept free\nfrom ads and trackers.\nIn turn, DeepCode gets exposed to many great developers that care about code\nquality and might become loyal customers. Also, they get recognized as an\nopen-source-friendly tech company, which is more important than ever if you're\ntrying to sell dev tools. Win-win!\n\n \n \n \n \nMarketing\nJakub and I both had started businesses before, but this was the first truly\nopen product we would build.\nPhase 1: Ship early πŸš€\nWe decided for a soft launch: deploy the site as early as possible and let the\ncrawlers index it. The fact that the page is statically rendered and follows\nsome basic SEO guidelines sure helped with improving our search engine rankings\nover time.\nPhase 2: Ask for feedback from your target audience πŸ’¬\nAfter we got some organic traffic and our first votes, we reached out to our\ndeveloper friends to test the page and vote on tools they know and love. This\nserved as an early validation, and we got some honest feedback, which helped us\ncatch the most blatant flaws.\nPhase 3: Prepare announcement post πŸ“\nWe wrote a blog post which, even if clickbaity, got the job done: Static\nAnalysis is Broken β€” Let's Fix\nIt! It\npretty much captures our frustration about the space and why building an open\nplatform is important. We could have done a better job explaining the technical\ndifferences between the different analysis tools, but that's for another day.\nPhase 4: Announce on social media πŸ”₯\nShortly before the official announcement, we noticed that the search\nfunctionality was broken (of course). Turns out, we hit the free quota limit on\nAlgolia a biiit earlier than expected. πŸ˜… No biggie: quick\nexchange with Algolia's customer support, and they moved us over to the\nopen-source plan (which we didn't know existed). We were back on track!\n\nSite note: Algolia customer support is top-notch. Responsive, tech-savvy,\nand helpful. Using Algolia turned out to be a great fit for our product.\nResponse times are consistently in the low milliseconds and the integration\nwith Gatsby was quick and easy.\n\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n We got quite a bit of buzz from that\ntweet: 63 retweets, 86 likes and counting\n \n \n \n \n \n\nClearly, everyone knew that we were asking for support here, but we are thankful\nfor every single one that liked and retweeted. It's one of these situations\nwhere having a network of like-minded people can help.\nAs soon as we were confident that the site wasn't completely broken, we set off\nto announce it on\nLobste.rs\n(2 downvotes),\n/r/SideProject\n(3 upvotes) and Hacker News (173\nupvotes, 57 comments). Social media is kind of unpredictable.\nIt helps to cater the message to each audience and stay humble, though.\nThe response from all of that marketing effort was nuts:\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Traffic on launch day\n \n \n \n \n \n\nPerhaps unsurprisingly, the Cloudflare edge workers didn't break a sweat.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Edge worker CPU time on Cloudflare\n \n \n \n \n \n\nMy boss Xoan Vilas even did a quick performance\nanalysis and he approved. (Thanks boss!)\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\nHigh fives all around!\n\n \n \n \n \nNow what?\nOf course, we'll add new features; of course, we have more plans for the future,\nyada yada yada. Instead, let's reflect on that milestone: a healthy little\nbusiness with no ads or trackers, solely carried by sponsors. πŸŽ‰\nFinally, I want you to look deep inside yourself and find your own little\nproduct to work on. It's probably\nright in front of your nose, and like myself, you've been putting it off for too\nlong. Well, not anymore! The next success story is yours. So go out and build\nthings.\nOh wait! ...before you leave, would you mind checking out\nanalysis-tools.dev and smashing that upvote button\nfor a few tools you like? Hey, and if you feel super generous today\n(or you have a fabulous employer that cares about open-source), why not check out\nour sponsorship page?\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Jakub and me in Vienna, Austria. I'm not actually that small.\n \n \n \n \n \n\n" }, { "title": "What Happened To Programming In The 2010s?", "url": "https://endler.dev/2020/review/", "body": "A while ago, I read an article titled "What Happened In The\n2010s" by Fred Wilson. The\npost highlights key changes in technology and business during the last ten\nyears. This inspired me to think about a much smaller topic: What Happened To\nProgramming In The 2010s?\n\n πŸš“ I probably forgot like 90% of what actually happened. Please\ndon't sue me.\nMy goal is to reflect on the past so that you can better predict the future.\n\n\n \n \n \n \nWhere To Start?\nFrom a mile-high perspective, programming is still the same as a decade ago:\n\nPunch program into editor\nFeed to compiler (or interpreter)\nBleep Boop πŸ€–\nReceive output\n\nBut if we take a closer look, a lot has changed around us.\nMany things we take for granted today didn't exist a decade ago.\n\n \n \n \n \nWhat Happened Before?\nBack in 2009, I wrote jQuery plugins, ran websites on\nshared hosting services, and uploaded content via\nFTP. Sometimes code was\ncopy-pasted from dubious forums, tutorials on blogs, or even hand-transcribed\nfrom books. Stack Overflow (which launched on\n15th of September 2008) was still in its infancy. Version control\nwas done with CVS or\nSVN β€” or not at all.\nI signed up for Github on 3rd of\nJanuary 2010. Nobody had even heard of a Raspberry\nPi (which only got released in\n2012).\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Source: xkcd #2324\n \n \n \n\n\n \n \n \n \nAn Explosion Of New Programming Languages\nThe last decade saw a vast number of new and exciting programming\nlanguages.\nCrystal, Dart, Elixir, Elm, Go, Julia, Kotlin, Nim, Rust, Swift, TypeScript\nall released their first stable version!\nEven more exciting: all of the above languages are developed in the open now, and the source code is\nfreely available on Github. That means, everyone can contribute to their development β€” a big testament to Open Source.\nEach of those languages introduced new ideas that were not widespread before:\n\nStrong Type Systems: Kotlin and Swift made optional null types\nmainstream, TypeScript brought types to JavaScript, Algebraic datatypes are\ncommon in Kotlin, Swift, TypeScript, and Rust.\nInteroperability: Dart compiles to JavaScript, Elixir interfaces with\nErlang, Kotlin with Java, and Swift with Objective-C.\nBetter Performance: Go promoted Goroutines and channels for easier\nconcurrency and impressed with a\nsub-millisecond Garbage Collector,\nwhile Rust avoids Garbage Collector overhead altogether thanks to ownership and borrowing.\n\nThis is just a short list, but innovation in the programming language field has\ngreatly accelerated.\n\n \n \n \n \nMore Innovation in Older Languages\nEstablished languages didn't stand still either. A few examples:\nC++ woke up from its long winter sleep and released C++11 after its last major\nrelease in 1998. It introduced numerous new features like Lambdas, auto\npointers, and range-based loops to the language.\nAt the beginning of the last decade, the latest PHP version was 5.3. We're at\n7.4 now. (We skipped 6.0, but I'm not ready to talk about it yet.) Along the\nway, it got over twice as fast. PHP is a truly modern programming language\nnow with a\nthriving ecosystem.\nHeck, even Visual Basic has tuples now. (Sorry, I couldn't resist.)\n\n \n \n \n \nFaster Release Cycles\nMost languages adopted a quicker release cycle. Here's a list for some popular languages:\nLanguageCurrent release cycle\nCirregular\nC#~ 12 months\nC++~ 3 years\nGo6 months\nJava6 months\nJavaScript (ECMAScript)12 months\nPHP12 months\nPython12 months\nRuby12 months\nRust6 weeks!\nSwift6 months\nVisual Basic .NET~ 24 months\n\n\n \n \n \n \nThe Slow Death Of Null\nClose to the end of the last decade, in a talk from 25thof August 2009,\nTony Hoare described the null pointer as his Billion Dollar\nMistake.\nA study by the Chromium project found that 70% of their serious security bugs were memory safety problems (same for Microsoft). Fortunately, the notion that our memory safety problem isn't bad coders\nhas finally gained some traction.\nMany mainstream languages embraced safer alternatives to null: nullable\ntypes, Option, and Result types. Languages like Haskell had these features\nbefore, but they only gained popularity in the 2010s.\n\n \n \n \n \nRevenge of the Type System\nClosely related is the debate about type\nsystems.\nThe past decade has seen type systems make their stage comeback; TypeScript,\nPython, and PHP (just to name a few) started to embrace type systems.\nThe trend goes towards type inference: add types to make your intent clearer for\nother humans and in the face of ambiguity β€” otherwise, skip them. Java,\nC++, Go, Kotlin, Swift, and Rust are popular examples with type inference support. I\ncan only speak for myself, but I think writing Java has become a lot more\nergonomic in the last few years.\n\n \n \n \n \nExponential Growth Of Libraries and Frameworks\nAs of today, npm hosts 1,330,634 packages. That's over a million\npackages that somebody else is maintaining for you. Add another 160,488 Ruby\ngems, 243,984 Python projects,\nand top it off with 42,547 Rust crates.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Number of packages for popular programming languages. Don't ask me what happened to npm in 2019.\n \n \n \n \n \n Source: Module Counts\n \n \n \n\nOf course, there's the occasional\nleftpad,\nbut it also means that we have to write less library code ourselves and can\nfocus on business value instead. On the other hand, there are more potential\npoints of failure, and auditing is difficult. There is also a large number of outdated\npackages. For a more in-depth discussion, I recommend the Census II report by\nthe Linux Foundation & Harvard [PDF].\nWe also went a bit crazy on frontend frameworks:\n\nAngular in 2010\nReact in 2013\nVue in 2014\nSvelte in 2016\n...and soon Yew?\n\n\n \n \n \n \nNo Free Lunch\nA review like this wouldn't be complete without taking a peek at Moore's Law.\nIt has held up surprisingly well in the last decade:\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Source: Wikipedia\n \n \n \n\nThere's a catch, though.\nLooking at single-core performance, the curve is flattening:\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Source: Standford University: The Future of Computing (video)\n \n \n \n\nThe new transistors prophesied by Moore don’t make our CPUs faster but instead\nadd other kinds of processing capabilities like more parallelism or hardware\nencryption.\nThere is no free lunch anymore. Engineers have to find new ways of making their\napplications faster, e.g. by embracing concurrent\nexecution.\nCallbacks, coroutines, and eventually async/await are becoming industry\nstandards.\nGPUs (Graphical Processing Units) became very powerful, allowing for massively\nparallel computations, which caused a renaissance of Machine Learning for practical use-cases:\n\nDeep learning becomes feasible, which leads to machine learning becoming\nintegral to many widely used software services and applications.\nβ€” Timeline of Machine Learning on Wikipedia\n\nCompute is ubiquitous, so in most cases, energy efficiency plays a more prominent role now than raw performance (at least for consumer devices).\n\n \n \n \n \nUnlikely Twists Of Fate\n\nMicrosoft is a cool kid now. It acquired Github, announced the Windows subsystem for Linux (which should really be called Linux Subsystem for Windows), open sourced\nMS-DOS and .NET.\nEven the Microsoft Calculator is now open source.\nIBM acquired Red Hat.\nLinus Torvalds apologized for his behavior, took time off.\nOpen source became the default for software development (?).\n\n\n \n \n \n \nLearnings\nIf you're now thinking: Matthias, you totally forgot X, then I brought\nthat point home. This is not even close to everything that happened. You'd\nroughly need a decade to talk about all of it.\nPersonally, I'm excited about the next ten years.\nSoftware eats the world β€” at an ever-faster pace.\n" }, { "title": "Tips for Faster Rust Compile Times", "url": "https://endler.dev/2020/rust-compile-times/", "body": "When it comes to runtime performance, Rust is one of the fastest guns in the\nwest. It is on par with the likes of C and\nC++\nand sometimes even surpasses them. Compile times, however? That's a different story.\n\n \n \n \n \nWhy Is Rust Compilation Slow?\nWait a sec, slow in comparison to what? That is, if you compare Rust with Go,\nthe Go compiler is doing a lot less work in general. For example, it lacks support for\ngenerics and macros. On top of that, the Go compiler was built from\nscratch\nas a monolithic toolchain consisting of both, the frontend and the backend (rather\nthan relying on, say, LLVM to take over the backend part,\nwhich is the case for Rust or Swift). This has advantages (more flexibility when\ntweaking the entire compilation process, yay) and disadvantages (higher overall maintenance cost\nand fewer supported architectures).\nIn general, comparing across different programming languages makes little sense\nand overall, the Rust compiler is legitimately doing a great job.\nThat said, above a certain project size, the compile times are... let's just say\nthey could be better.\n\n \n \n \n \nWhy Bother?\nAccording to the Rust 2019\nsurvey, improving\ncompile times is #4 on the Rust wishlist:\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n Rust Survey results 2019. (Obligatory xkcd.)\n \n \n \n \n \n\n\n \n \n \n \nCompile-Time vs Runtime Performance\n\nAs is often cautioned in debates among their designers, programming language\ndesign is full of tradeoffs. One of those fundamental tradeoffs is runtime\nperformance vs. compile-time performance, and the Rust team nearly always (if\nnot always) chose runtime over compile-time.\nβ€” Brian Anderson\n\nOverall, there are a few features and design decisions that limit Rust\ncompilation speed:\n\nMacros: Code generation with macros can be quite expensive.\nType checking\nMonomorphization: this is the process of generating specialized versions\nof generic functions. E.g., a function that takes an Into<String> gets\nconverted into one that takes a String and one that takes a &str.\nLLVM: that's the default compiler backend for Rust, where a lot of the\nheavy-lifting (like code-optimizations) takes place. LLVM is notorious for\nbeing slow.\nLinking: Strictly speaking, this is not part of compiling but happens\nright after. It "connects" your Rust binary with the system libraries. cargo\ndoes not explicitly mark the linking step, so many people add it to the\noverall compilation time.\n\nIf you're interested in all the gory details, check out this blog\npost by Brian\nAnderson.\n\n \n \n \n \nUpstream Work\nMaking the Rust compiler faster is an ongoing process, and many fearless people\nare working on\nit.\nThanks to their hard work, compiler speed has improved 30-40% across the board\nyear-to-date, with some projects seeing up to 45%+ improvements.\nOn top of that, Rust tracks compile regressions on a website dedicated to\nperformance\nWork is also put into optimizing the LLVM\nbackend. Rumor\nhas it that there's still a lot of low-hanging fruit. πŸ‡\n\n \n \n \n \nWhat You Can Improve Right Away\nUpstream work takes time, but what if you have a performance problem right\nnow and can't wait? Well, all hope is not lost! Below is a list of tips and\ntricks on how to make your Rust project compile faster today. They are roughly\nordered by\npracticality, so start at the top and work your way down until you're happy and your compiler goes brrrrrrr.\n\n \n \n \n \nUse cargo check Instead Of cargo build\nMost of the time, you don't even have to compile your project at all; you just\nwant to know if you messed up somewhere. Whenever you can, skip compilation\naltogether. What you need instead is laser-fast code linting, type- and\nborrow-checking.\nFor that, cargo has a special treat for you: ✨ cargo check ✨. Consider the\ndifferences in the number of instructions between cargo check on the left and\ncargo debug in the middle. (Pay attention to the different scales.)\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Speedup factors: check 1, debug 5, opt 20\n \n \n \n \n \n\nA sweet trick I use is to run it in the background with cargo watch. This way, it will cargo check\nwhenever you change a file.\n⭐ Pro-tip: Use cargo watch -c to clear the screen before every run.\n\n \n \n \n \nUse Rust Analyzer Instead Of Rust Language Server (RLS)\nAnother quick way to check if you set the codebase on fire is to use a "language\nserver". That's basically a "linter as a service", that runs next to your\neditor.\nFor a long time, the default choice here was\nRLS, but lately, folks moved over to\nrust-analyzer, because it's\nmore feature-complete and way more snappy. It supports all major IDEs.\nSwitching to that alone might save your day.\n\n ✨WOW! This article is getting quite a bit of traction lately.βœ¨β€\nI don't run ads on this website, but if this information is helping you, please\nconsider sponsoring me on Github.\nThis allows me to write content like this.\nAlso, if you're interested in hands-on Rust consulting, pick a date from my\ncalendar and we can talk about how I can help .\n\n\n \n \n \n \nRemove Unused Dependencies\nSo let's say you tried all of the above and find that compilation is still slow.\nWhat now?\nDependencies sometimes become obsolete thanks to refactoring. From time to time\nit helps to check if all of them are still needed to save compile time.\nIf this is your own project (or a project you like to contribute to), do a quick\ncheck if you can toss anything with\ncargo-udeps:\n\ncargo install cargo-udeps && cargo +nightly udeps\n\n \n \n \n \nUpdate Remaining Dependencies\nNext, update your dependencies, because they themselves could have tidied up\ntheir dependency tree lately.\nTake a deep dive with\ncargo-outdated or cargo tree\n(built right into cargo itself) to find any outdated dependencies. On top of\nthat, use cargo audit to get\nnotified about any vulnerabilities which need to be addressed, or deprecated\ncrates which need a replacement.\nHere's a nice workflow that I learned from /u/oherrala on\nReddit:\n\nRun cargo update to update to the latest semver\ncompatible version.\nRun cargo outdated -wR to find newer, possibly incompatible dependencies.\nUpdate those and fix code as needed.\nFind duplicate versions of a dependency and figure out\nwhere they come from: cargo tree --duplicate shows dependencies which come in multiple versions.\n(Thanks to /u/dbdr for pointing this out.)\n\n⭐ Pro-tip: Step 3 is a great way to contribute back to the community! Clone\nthe repository and execute steps 1 and 2. Finally, send a pull request to the\nmaintainers.\n\n \n \n \n \nReplace Heavy Dependencies\nFrom time to time, it helps to shop around for more lightweight alternatives to\npopular crates.\nAgain, cargo tree is your friend here to help you understand which of your\ndependencies are quite heavy: they require many other crates, cause\nexcessive network I/O and slow down your build. Then search for lighter\nalternatives.\nAlso, cargo-bloat has a --time\nflag that shows you the per-crate build time. Very handy!\nHere are a few examples:\n\nUsing serde? Check out\nminiserde and maybe even\nnanoserde.\nreqwests is quite heavy. Maybe try\nattohttpc or ureq, which are more lightweight.\ntokio dragging you down? How about smol?\n(Edit: This won't help much with build times. More info in this discussion on Reddit)\nSwap out clap with pico-args if you only need basic argument parsing.\n\nHere's an example where switching crates reduced compile times from 2:22min to\n26\nseconds.\n\n \n \n \n \nUse Cargo Workspaces\nCargo has that neat feature called workspaces, which allow you to split one\nbig crate into multiple smaller ones. This code-splitting is great for avoiding\nrepetitive compilation because only crates with changes have to be recompiled.\nBigger projects like\nservo and\nvector\nare using workspaces heavily to slim down compile times.\nLearn more about workspaces here.\n\n \n \n \n \nCombine All Integration Tests In A Single Binary\nHave any integration tests? (These are the ones in your tests\nfolder.)\nDid you know that the Rust compiler will create a binary for every single one of them?\nAnd every binary will have to be linked individually.\nThis can take most of your build time because linking is slooow. 🐒\nThe reason is that many system linkers (like ld) are single\nthreaded.\n\n πŸ‘¨β€πŸ³οΈπŸ’‘β€οΈ A linker is a tool that combines the\noutput of a compiler and mashes that into one executable you can run.\n\nTo make the linker's job a little easier, you can put all your tests in one\ncrate. (Basically create a main.rs in your test folder and add your\ntest files as mod in there.)\nThen the linker will go ahead and build a single binary only. Sounds nice, but\ncareful: it's still a trade-off as you'll need to expose your internal types and\nfunctions (i.e. make them pub).\nMight be worth a try, though because a recent benchmark revealed a 1.9x\nspeedup for one project.\nThis tip was brought to you by Luca Palmieri,\nLucio Franco, and Azriel\nHoh. Thanks!\n\n \n \n \n \nDisable Unused Features Of Crate Dependencies\n⚠️ Fair warning: it seems that switching off features doesn't always improve\ncompile time. (See tikv's experiences\nhere.)\nCheck the feature flags of your dependencies. A lot of library maintainers take\nthe effort to split their crate into separate features that can be toggled off\non demand. Maybe you don't need all the default functionality from every crate?\nFor example, tokio has a ton of\nfeatures\nthat you can disable if not needed.\nA quick way to list all features of a crate is\n[cargo-feature-set](https://github.com/badboy/cargo-feature-set).\nAdmittedly, features are not very discoverable at the\nmoment because there is\nno standard way to document them, but we'll get there eventually.\n\n \n \n \n \nUse A Ramdisk For Compilation\nWhen starting to compile heavy projects, I noticed that I was throttled on I/O.\nThe reason was that I kept my projects on a measly HDD. A more performant\nalternative would be SSDs, but if that's not an option, don't throw in the\nsponge just yet.\nRamdisks to the rescue! These are like "virtual harddisks" that live in system\nmemory.\nUser moschroe_de shared the\nfollowing snippet over on\nReddit,\nwhich creates a ramdisk for your current Rust project (on Linux):\n\nmkdir -p target && \\\nsudo mount -t tmpfs none ./target && \\\ncat /proc/mounts | grep "$(pwd)" | sudo tee -a /etc/fstab\n\nOn macOS, you could probably do something similar with this\nscript. I haven't tried that myself,\nthough.\n\n \n \n \n \nCache Dependencies With sccache\nAnother neat project is sccache by\nMozilla, which caches compiled crates to avoid repeated compilation.\nI had this running on my laptop for a while, but the benefit was rather\nnegligible, to be honest. It works best if you work on a lot of independent\nprojects that share dependencies (in the same version). A common use-case is\nshared build servers.\n\n \n \n \n \nCranelift – The Alternative Rust Compiler\nLately, I was excited to hear that the Rust project is using an alternative\ncompiler that runs in parallel with rustc for every CI build:\nCranelift, also called\nCG_CLIF.\nHere is a comparison between rustc and Cranelift for some popular crates (blue\nmeans better):\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n LLVM compile time comparison between rustc and cranelift in favor of cranelift\n \n \n \n \n \n\nSomewhat unbelieving, I tried to compile\nvector with both compilers.\nThe results were astonishing:\n\nRustc: 5m 45s\nCranelift: 3m 13s\n\nI could really notice the difference! What's cool about this is that it creates\nfully working executable binaries. They won't be optimized as much, but they are\ngreat for local development.\nA more detailed write-up is on Jason Williams'\npage, and the\nproject code is on Github.\n\n \n \n \n \nSwitch To A Faster Linker\n\nThe thing that nobody seems to target is linking time. For me, when using\nsomething with a big dependency tree like Amethyst, for example linking time\non my fairly recent Ryzen 7 1700 is ~10s each time, even if I change only some\nminute detail only in my code. β€” /u/Almindor on\nReddit\n\nAccording to the official documentation, "LLD is a\nlinker from the LLVM project that is a drop-in replacement for system linkers\nand runs much faster than them. [..] When you link a large program on a\nmulticore machine, you can expect that LLD runs more than twice as fast as the\nGNU gold linker. Your mileage may vary, though."\nIf you're on Linux you can switch to lld like\nso:\n\n[target.x86_64-unknown-linux-gnu]\nrustflags = [\n "-C", "link-arg=-fuse-ld=lld",\n]\n\nA word of caution: lld might not be working on all platforms yet. At least on\nmacOS, Rust support seems to be broken at the moment, and the work on fixing it\nhas stalled (see\nrust-lang/rust#39915).\nUpdate: I recently learned about another linker called mold, which claims a massive 12x performance bump over lld. Compared to GNU gold, it's said to be more than 50x. Would be great if anyone could verify and send me a message.\nUpdate II: Aaand another one called zld, which is a drop-in replacement for Apple's ld linker and is targeting debug builds. [Source]\nWhich one you want to choose depends on your requirements. Which platforms do\nyou need to support? Is it just for local testing or for production usage?\n\n \n \n \n \nFaster Incremental Debug Builds On Macos\nRust 1.51 added an interesting flag for faster incremental debug builds on\nmacOS. It can make debug builds up to seconds faster (depending on your use-case).\nJust add this to your Cargo.toml:\n\n[profile.dev]\nsplit-debuginfo = "unpacked"\n\nSome engineers report that this flag alone reduces compilation times on macOS by 70%.\nThe flag might become the standard for macOS soon. It is already the default\non nightly.\n\n \n \n \n \nTweak More Codegen Options / Compiler Flags\nRust comes with a huge set of settings for code\ngeneration. It can help to\nlook through the list and tweak the parameters for your project.\nThere are many gems in the full list of codegen\noptions. For inspiration,\nhere's bevy's config for faster\ncompilation.\n\n \n \n \n \nProfile Compile Times\nIf you like to dig deeper, Rust compilation can be profiled with cargo rustc -- -Zself-profile.\nThe resulting trace file can be visualized with a flamegraph or the Chromium\nprofiler:\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Image of Chrome profiler with all crates\n \n \n \n \n \n Source: Rust Lang Blog\n \n \n \n\nThere's also a cargo -Z timings\nfeature that gives some information about how long each compilation step takes,\nand tracks concurrency information over time.\nAnother golden one is\ncargo-llvm-lines, which shows\nthe number of lines generated and objects copied in the LLVM backend:\n\n$ cargo llvm-lines | head -20\n\n Lines Copies Function name\n ----- ------ -------------\n 30737 (100%) 1107 (100%) (TOTAL)\n 1395 (4.5%) 83 (7.5%) core::ptr::drop_in_place\n 760 (2.5%) 2 (0.2%) alloc::slice::merge_sort\n 734 (2.4%) 2 (0.2%) alloc::raw_vec::RawVec<T,A>::reserve_internal\n 666 (2.2%) 1 (0.1%) cargo_llvm_lines::count_lines\n 490 (1.6%) 1 (0.1%) <std::process::Command as cargo_llvm_lines::PipeTo>::pipe_to\n 476 (1.5%) 6 (0.5%) core::result::Result<T,E>::map\n 440 (1.4%) 1 (0.1%) cargo_llvm_lines::read_llvm_ir\n 422 (1.4%) 2 (0.2%) alloc::slice::merge\n 399 (1.3%) 4 (0.4%) alloc::vec::Vec<T>::extend_desugared\n 388 (1.3%) 2 (0.2%) alloc::slice::insert_head\n 366 (1.2%) 5 (0.5%) core::option::Option<T>::map\n 304 (1.0%) 6 (0.5%) alloc::alloc::box_free\n 296 (1.0%) 4 (0.4%) core::result::Result<T,E>::map_err\n 295 (1.0%) 1 (0.1%) cargo_llvm_lines::wrap_args\n 291 (0.9%) 1 (0.1%) core::char::methods::<impl char>::encode_utf8\n 286 (0.9%) 1 (0.1%) cargo_llvm_lines::run_cargo_rustc\n 284 (0.9%) 4 (0.4%) core::option::Option<T>::ok_or_else\n\n \n \n \n \nAvoid Procedural Macro Crates\nProcedural macros are the hot sauce of Rust development: they burn through CPU\ncycles so use with care (keyword: monomorphization).\nUpdate: Over on Twitter\nManish pointed out that "the reason proc macros\nare slow is that the (excellent) proc macro infrastructure – syn and friends – are slow to compile. Using proc macros themselves does not have a huge impact on compile times."\n(This might change in the future.)\nManish goes on to say\n\nThis basically means that if you use one proc macro, the marginal compile time\ncost of adding additional proc macros is insignificant. A lot of people end up\nneeding serde in their deptree anyway, so if you are forced to use serde, you\nshould not care about proc macros.\nIf you are not forced to use serde, one thing a lot of folks do is have\nserde be an optional dependency so that their types are still serializable\nif necessary.\n\nIf you heavily use procedural macros in your project (e.g., if you use serde),\nyou can try to sidestep their impact on compile times with\nwatt, a tool that offloads macro compilation\nto Webassembly.\nFrom the docs:\n\nBy compiling macros ahead-of-time to Wasm, we save all downstream users of the\nmacro from having to compile the macro logic or its dependencies themselves.\nInstead, what they compile is a small self-contained Wasm runtime (~3 seconds,\nshared by all macros) and a tiny proc macro shim for each macro crate to hand\noff Wasm bytecode into the Watt runtime (~0.3 seconds per proc-macro crate you\ndepend on). This is much less than the 20+ seconds it can take to compile\ncomplex procedural macros and their dependencies.\n\nNote that this crate is still experimental.\n(Oh, and did I mention that both, watt and cargo-llvm-lines were built by\nDavid Tolnay, who is a frickin' steamroller of an\nengineer?)\n\n \n \n \n \nCompile On A Beefy Machine\nOn portable devices, compiling can drain your battery and be slow. To avoid\nthat, I'm using my machine at home, a 6-core AMD FX 6300 with 12GB RAM, as a\nbuild machine. I can use it in combination with Visual Studio Code Remote\nDevelopment.\nIf you don't have a dedicated machine yourself, you can compile in the cloud\ninstead.\nGitpod.io is superb for testing a cloud build as they\nprovide you with a beefy machine (currently 16 core Intel Xeon 2.30GHz, 60GB\nRAM) for free during a limited period. Simply add https://gitpod.io/# in\nfront of any Github repository URL.\nHere is an example for one of my Hello Rust episodes.\nWhen it comes to buying dedicated hardware,\nhere are some tips. Generally, you should get a proper multicore CPU like an AMD\nRyzen Threadripper plus at least 32 GB of RAM.\n\n \n \n \n \nDownload ALL The Crates\nIf you have a slow internet connection, a big part of the initial build\nprocess is fetching all those shiny crates from crates.io. To mitigate that,\nyou can download all crates in advance to have them cached locally.\ncriner does just that:\n\ngit clone https://github.com/the-lean-crate/criner\ncd criner\ncargo run --release -- mine\n\nThe archive size is surprisingly reasonable, with roughly 50GB of required disk\nspace (as of today).\n\n \n \n \n \nBonus: Speed Up Rust Docker Builds 🐳\nBuilding Docker images from your Rust code?\nThese can be notoriously slow, because cargo doesn't support building only a\nproject's dependencies yet, invalidating the Docker cache with every build if you\ndon't pay attention.\ncargo-chef to the\nrescue! ⚑\n\ncargo-chef can be used to fully leverage Docker layer caching, therefore\nmassively speeding up Docker builds for Rust projects. On our commercial\ncodebase (~14k lines of code, ~500 dependencies) we measured a 5x speed-up: we\ncut Docker build times from ~10 minutes to ~2 minutes.\n\nHere is an example Dockerfile if you're interested:\n\n# Step 1: Compute a recipe file\nFROM rust as planner\nWORKDIR app\nRUN cargo install cargo-chef\nCOPY . .\nRUN cargo chef prepare --recipe-path recipe.json\n\n# Step 2: Cache project dependencies\nFROM rust as cacher\nWORKDIR app\nRUN cargo install cargo-chef\nCOPY --from=planner /app/recipe.json recipe.json\nRUN cargo chef cook --release --recipe-path recipe.json\n\n# Step 3: Build the binary\nFROM rust as builder\nWORKDIR app\nCOPY . .\n# Copy over the cached dependencies from above\nCOPY --from=cacher /app/target target\nCOPY --from=cacher /usr/local/cargo /usr/local/cargo\nRUN cargo build --release --bin app\n\n# Step 4:\n# Create a tiny output image.\n# It only contains our final binary.\nFROM rust as runtime\nWORKDIR app\nCOPY --from=builder /app/target/release/app /usr/local/bin\nENTRYPOINT ["./usr/local/bin/app"]\n\ncargo-chef can help speed up\nyour continuous integration with Github Actions or your deployment process to Google\nCloud.\n\n \n \n \n \nDrastic Measures: Overclock Your CPU? πŸ”₯\n\n \n ⚠️ Warning: You can damage your hardware if you don't know what you are doing.\n Proceed at your own risk.\n \nHere's an idea for the desperate. Now I don't recommend that to everyone, but\nif you have a standalone desktop computer with a decent CPU, this might be a way\nto squeeze out the last bits of performance.\nEven though the Rust compiler executes a lot of steps in parallel,\nsingle-threaded performance is still quite relevant.\nAs a somewhat drastic measure, you can try to overclock your CPU. Here's a\ntutorial for my processor. (I owe\nyou some benchmarks from my machine.)\n\n\n \n \n \n \nHelp Others: Upload Leaner Crates For Faster Build Times\ncargo-diet helps you build\nlean crates that significantly reduce download size (sometimes by 98%). It might\nnot directly affect your own build time, but your users will surely be thankful. 😊\n\n \n \n \n \nMore Resources\n\nThe Rust Perf\nBook has a\nsection on compile times.\nTons of articles on performance on Read\nRust.\n8 Solutions for Troubleshooting Your Rust Build\nTimes\nis a great article by Dotan Nahum that I fully agree with.\nImproving the build times of a bigger Rust project (lemmy) by\n30%.\narewefastyet measures how long the Rust compiler\ntakes to compile common Rust programs.\n\n\n \n \n \n \nWhat's Next?\nPhew! That was a long list. πŸ˜… If you have any additional tips, please let me know.\nIf compiler performance is something you're interested in, why not collaborate\non a tool to see what user code is causing rustc to use lots of\ntime?\nAlso, once you're done optimizing your build times, how about optimizing\nruntimes next? My friend Pascal Hertleif has a\nnice article on that.\n" }, { "title": "Gravity", "url": "https://endler.dev/2020/gravity/", "body": "Here's a test to show your age:\nDo you still remember that funny JavaScript gravity effect, which Google used on their homepage ten years ago? This one?\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\nI wanted to have some fun and integrated it into a website I was building.\nUnfortunately, it didn't work out-of-the-box.\nIt choked on some DOM elements that were not strictly classes (like SVG elements).\nSo, in good hacker fashion, I quickly patched up the script (it's just a three-line change), and now it's back to its former glory.\nTest it here! (Caution: you'll have to reload the page after that. 😏)\nApply Gravity\n\n var myLink = document.getElementById('gravity');\n\n myLink.onclick = function(){\n\n var script = document.createElement(\"script\");\n script.type = \"text/javascript\";\n script.src = \"gravity.js\"; \n document.getElementsByTagName(\"head\")[0].appendChild(script);\n return false;\n }\n\nAnyway, feel free to add it to your own sites and have some fun.\nIt's also great to prank your friends.\nSimply add that single line to any website and weeee!\n\n<script\n type="text/javascript"\n src="https://endler.dev/2020/gravity/gravity.js"\n></script>\n\nSometimes I miss those simple times of the early web...\n" }, { "title": "Hacker Folklore", "url": "https://endler.dev/2020/folklore/", "body": "Some computer terms have a surprising legacy. Many of them are derived from\nlong-obsolete technologies. This post tries to dust off the exciting history of\nsome\nof these terms that we use every day but aren't quite sure about their origins.\nMost of the content comes from sources like Wikipedia (with reference where\nappropriate), but the explanations are hard to hunt down if you don't know what\nyou're looking for.\nThis is a living document, and I'm planning to update it in case of reader\nsubmissions.\n\n \n \n \n \nBike-Shedding\nToday's meaning: A pointless discussion about trivial issues.\nThe term bike-shed effect or bike-shedding was coined as a metaphor to\nilluminate the law of triviality; it was popularised in the Berkeley Software\nDistribution community by the Danish computer developer Poul-Henning Kamp in\n1999 on the FreeBSD mailing list and has\nspread from there to the whole software industry.\nThe concept was first presented as a corollary of his broader "Parkinson's law"\nspoof of management. He dramatizes this "law of triviality" with the example of\na committee's deliberations on an atomic reactor, contrasting it to\ndeliberations on a bicycle shed. As he put it: "The time spent on any item of\nthe agenda will be in inverse proportion to the sum of money involved." A\nreactor is so vastly expensive and complicated that an average person cannot\nunderstand it, so one assumes that those who work on it understand it. On the\nother hand, everyone can visualize a cheap, simple bicycle shed, so planning one\ncan result in endless discussions because everyone involved wants to add a touch\nand show personal contribution.\nReference - Wikipedia: Law of Triviality\n\n \n \n \n \nBoilerplate\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n An old machine that bended steel\nplates to water boilers.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\nToday's meaning: A chunk of code that is copied over and over again with little\nor no changes made to it in the process.\nBoiler plate originally referred to the rolled steel used to make water\nboilers but is used in the media to refer to hackneyed or unoriginal writing.\nThe term refers to the metal printing plates of pre-prepared text such as\nadvertisements or syndicated columns that were distributed to small, local\nnewspapers. These printing plates came to be known as 'boilerplates' by analogy.\nOne large supplier to newspapers of this kind of boilerplate was the Western\nNewspaper Union, which supplied "ready-to-print stories [which] contained\nnational or international news" to papers with smaller geographic footprints,\nwhich could include advertisements pre-printed next to the conventional content.\nReferences:\n\nWikipedia\nStack Overflow\nStackExchange - English Language &\nUsage\n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n The man in the foreground is holding\na rounded printing plate. Plates like this were provided by companies such as\nWestern Newspaper Union to many smaller newspapers.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n \n \n \nBug\nToday's meaning: A defect in a piece of code or hardware.\nThe origins are unknown!\nContrary to popular belief it predates the bug\nfound by Grace Hopper in the Mark II computer.\nThe term was used by engineers way before that; at least since the 1870s.\nIt predates electronic computers and computer software.\nThomas Edison used the term "bug" in his notes.\nReference\n\n \n \n \n \nCarriage Return and Line Feed\nToday's meaning: Set the cursor to the beginning of the next line.\nThese two terms were adopted from typewriters.\nThe carriage holds the paper and is moving from left to right to advance the\ntyping position as the keys are pressed. It "carries" the paper with it. The\ncarriage return is the operation when the carriage gets moved into its original\nposition on the very left end side of the paper.\nSimply returning the carriage to the left is not enough to start with a new\nline, however. The carriage would still be on the same line than before β€”\njust at the beginning of the line. To go to a new line, a line feed was\nneeded. It would move the paper inside the typewriter up by one line.\nThese two operations β€” carriage return (CR) and line feed (LF) β€”\nwere commonly done at once by pushing the carriage return lever.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A mechanical typewriter. The lever for the carriage return is\non the outer left side.\n \n \n \n \n \n Source: Source:\npiqsels\n \n \n \n\n\nOn Unix systems (like Linux or macOS), a \\n still stands for a\nline feed (ASCII symbol: LF) or newline.\nOn CP/M, DOS, and Windows, \\r\\n is used, where \\r stands for\ncarriage return and \\n stands for line feed (CR+LF).\nReference\n\nHere is an old video that shows the basic mechanics of carriage return and\nline-feed:\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\n\n \n \n \n \nCommand key symbol (⌘)\nToday's meaning: A meta-key available on Apple computers to provide additional\nkeyboard combinations.\nDirectly quoting Wikipedia (emphasis mine):\nThe ⌘ symbol came into the Macintosh project at a late stage. The development\nteam originally went for their old Apple key, but Steve Jobs found it\nfrustrating when "apples" filled up the Mac's menus next to the key commands,\nbecause he felt that this was an over-use of the company logo. He then opted for\na different key symbol. With only a few days left before deadline, the team's\nbitmap artist Susan Kare started researching for the Apple logo's successor. She\nwas browsing through a symbol dictionary when she came across the\ncloverleaf-like symbol, commonly used in Nordic countries as an indicator of\ncultural locations and places of interest (it is the official road sign for\ntourist attraction in Denmark, Finland, Iceland, Norway, and Sweden and the\ncomputer key has often been called Fornminne β€” ancient monument β€” by Swedish Mac\nusers and SevΓ¦rdighedstegn by Danish users). When she showed it to the rest of\nthe team, everyone liked it, and so it became the symbol of the 1984 Macintosh\ncommand key. Susan Kare states that it has since been told to her that the\nsymbol had been picked for its Scandinavian usage due to its resembling the\nshape of a square castle with round corner towers as seen from above looking\ndown, notably Borgholm Castle.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Norwegian Severdighet road sign\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Aearial view of Borgholm Castle, which could have been the model for the symbol \n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\nReferences:\n\nWikipedia: Command Key\nCult of Mac: What Are The Mac’s Command ⌘ And Option βŒ₯ Symbols Supposed To\nRepresent?\n\n\n \n \n \n \nCore Dump\nToday's meaning: Retrieving a snapshot of a (crashed) program's state by\nstoring all of its memory for offline analysis.\nThe name comes from magnetic core\nmemory, which is an early\nstorage mechanism based on a grid of toroid magnets. It has since become\nobsolete, but the term is still used today for getting a snapshot of a computer\nprocess. Reference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A 32 x 32 core memory plane storing\n1024 bits (or 128 bytes) of data. The first core dumps were printed on paper, which sounds reasonable given these small amounts of bytes.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n \n \n \nCursor\nToday's meaning: a visual cue (such as a flashing vertical line) on a video display that indicates position (as for data entry). Merriam-Webster\nCursor is Latin for runner. A cursor is the name given to the transparent\nslide engraved with a hairline that is used for marking a point on a slide rule.\nThe term was then transferred to computers through analogy.\nReference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Source: A December 1951 advertisement for the\nIBM 604 Electronic Calculating Punch that was first produced in 1948. The\nadvertisement claims the IBM 604 can do the work of 150 engineers with slide\nrules. The cursor (or runner) is the transparent part in the middle of the\nslide.\n \n \n \n\n\n \n \n \n \nDashboard\nToday's meaning: A user interface that provides a quick overview of a system's\nstatus.\nOriginally a plank of wood at the front of a horse-drawn carriage to protect the\ndriver from mud 'dashed' backward by a horses hooves.\nWhen automobiles were manufactured, the board in front of the driver was given\nthe same name. That was the logical place to put the necessary gauges so the\ndriver could see them easily. In time, the term became more associated with the\nreadouts than the protection it offered.\nReference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A dashboard of a horse carriage.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n \n \n \nFirewall\nToday's meaning: A network security system that establishes a barrier between a trusted internal network and an untrusted external network, such as the Internet.\nFire walls are used mainly in terraced houses, but also in individual residential buildings. They prevent fire and smoke from spreading to another part of the building in the event of a fire. Large fires can thus be prevented. The term is used in computing since the 80s.\nReference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Firewall residential construction, separating the building into two separate residential units, and fire areas.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n \n \n \nFirmware\nToday's meaning: A class of computer software that provides the low-level\ncontrol for the device's specific hardware and closely tied to the hardware it runs on.\nAscher\nOpler\ncoined the term firmware in a 1967 Datamation article. As originally used,\nfirmware contrasted with hardware (the CPU itself) and software (normal\ninstructions executing on a CPU). It existed on the boundary between hardware\nand software; thus the name "firmware". The original article is available on the\nInternet\nArchive.\nReference\n\n \n \n \n \nFoo and Bar\nToday's meaning: Common placeholder variable names.\nOriginally the term might come from the military term FUBAR.\nThere are a few variations, but a common meaning is FUBAR: "f***ed up beyond all recognition".\nThe use of foo in a programming context is generally credited to the Tech\nModel Railroad Club (TMRC) of MIT from circa 1960. In the complex model system,\nthere were scram switches located at numerous places around the room that could\nbe thrown if something undesirable was about to occur, such as a train going\nfull-bore at an obstruction.\nThe way I understood it was that they literally had emergency buttons labeled\nfoo for lack of a better name.\nMaybe related to the original military meaning of FUBAR to indicate that something is going very very wrong.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A scram switch (button), that could be\npressed to prevent inadvertent operation. Maybe the TMRC had buttons labeled foo instead\n \n \n \n \n \n Source: Source Wikimedia\nCommons\n \n \n \n\nReferences:\n\nWikipedia\nStack Overflow.\n\n\n \n \n \n \nFreelancer\nToday's meaning: A self-employed person, which is not committed to a particular employer long-term.\nThe term first appears in the novel Ivanhoe by Sir Walter Scott. (The novel also had a lasting influence on the Robin Hood legend.)\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Cover of a Classic Comics book\n \n \n \n \n \n Source: Wikimedia Commons\n \n \n \n\nIn it, a Lord offers his paid army of 'free lances' to King Richard:\n\nI offered Richard the service of my Free Lances, and he refused them\nβ€” I will lead them to Hull, seize on shipping, and embark for Flanders;\nthanks to the bustling times, a man of action will always find employment.\n\nTherefore, a "free lancer" is someone who fights for whoever pays the most.\nFree does not mean "without pay", but refers to the additional freedom to work for any employer. Reference\n\n \n \n \n \nLog / Logfile\nToday's meaning: A file that records events of a computer program or system.\nSailors used so-called log lines to measure the speed of their ship. A flat\npiece of wood (the log) was attached to a long rope. The log had regularly\nspaced knots in it. As the log would drift away, the sailors would count the\nnumber of knots that went out in a fixed time interval, and this would be the\nship's speed β€” in knots.\nThe ship's speed was important for navigation, so the sailors noted it down in a book, aptly called the log book, together with other information to establish the position of the ship more accurately, like landmark sightings and weather events. Later, additional information, more generally concerning the ship, was added β€” or logged β€” such as harbor fees and abnormal provision depletion.\nReference.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Sailors measuring ship speed with a\nlog line\n \n \n \n \n \n Source: The Pilgrims & Plymouth Colony:1620 by Duane A.\nCline\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n The parts of a log-line\n \n \n \n \n \n Source: The\nPilgrims & Plymouth Colony:1620 by Duane A.\nCline\n \n \n \n\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Page from the log-file of the British\nWinchelsea. The second column denotes the number of knots measured with the\nlog-line, which indicates the ship's speed\n \n \n \n \n \n Source: Navigation and Logbooks\nin the Age of Sail by Peter\nReaveley\n \n \n \n\n\n \n \n \n \nPatch\nToday's meaning: A piece of code that can be applied to fix or improve a\ncomputer program.\nIn the early days of computing history, if you made a programming mistake, you'd\nhave to fix a paper tape or a punched card by putting a patch on top of a hole.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A program tape with physical patches used\nto correct punched holes by covering them.\n \n \n \n \n \n Source: Smithsonian Archives\nCenter\n \n \n \n\n\n \n \n \n \nPing\nToday's meaning: A way to check the availability and response time of a computer over the network.\nPing is a terminal program originally written by Mike Muuss in 1983 that is included in\nevery version of UNIX, Windows, and macOS. He named it "after the sound that a\nsonar makes, inspired by the whole principle of echo-location. [...] ping uses\ntimed IP/ICMP ECHO_REQUEST and ECHO_REPLY packets to probe the "distance" to the\ntarget machine." The reference is\nwell worth a read.\n\n \n \n \n \nPixel\nToday's meaning: The smallest controllable element of a picture represented on the screen.\nThe word pixel is a combination of pix (from "pictures", shortened to "pics") and el (for "element").\nSimilarly, voxel is a volume element and texel is a texture element.\nReference\n\n \n \n \n \nShell\nToday's meaning: An interactive, commonly text-based runtime to interact with a\ncomputer system.\nThe inventor of the term, Louis Pouzin, does not give an explanation for the\nname in his essay The Origins of the\nShell. It can however be traced back to\nUnix' predecessor Multics. It is described in the Multics\nglossary like so:\n\n[The shell] is passed a command line for execution by the listener.\n\nThe The New Hacker's Dictionary, (also known as the Jargon File) by Eric S.\nRaymond contains the\nfollowing:\n\nHistorical note: Apparently, the original Multics shell (sense 1) was so\ncalled because it was a shell (sense 3);\n\nwhere sense 3 refers to\n\nA skeleton program, created by hand or by another program (like, say, a parser\ngenerator), which provides the necessary incantations to set up some task and\nthe control flow to drive it (the term driver is sometimes used synonymously).\nThe user is meant to fill in whatever code is needed to get real work done.\nThis usage is common in the AI and Microsoft Windows worlds, and confuses Unix\nhackers.\n\nUnfortunately, the book does not provide any evidence to back up this claim.\nI like the (possibly historically incorrect) analogy to a nut with the shell\nbeing on the outside, protecting the kernel.\nReference\n\n \n \n \n \nSlab allocator\nToday's meaning: An efficient memory allocation technique, which reuses\nprevious allocations.\nSlab allocation was invented by John\nBonwick\n(Note: PDF file) in 1994 and has since been used by services like\nMemcached and the Linux Kernel.\n\nWith slab allocation, a cache for a certain type or size of data object has a\nnumber of pre-allocated "slabs" of memory; within each slab there are memory\nchunks of fixed size suitable for the objects.\n(Wikpedia)\n\nThe name slab comes from a teenage friend of Bonwick. He tells the\nstory on the Oracle blog:\nWhile watching TV together, a commercial by Kellogg's came on with the tag line,\n"Can you pinch an inch?"\n\nThe implication was that you were overweight if you could pinch more than an\ninch of fat on your waist β€” and that hoovering a bowl of corn flakes would\nhelp.\n\n\nWithout missing a beat, Tommy, who weighed about 250 pounds, reached for his\nmidsection and offered his response: "Hell, I can grab a slab!"\n\nA decade later, Bonwick remembered that term when he was looking for a word to\ndescribe the allocation of a larger chunk of memory.\nHere is the original Kellogg's advertisement:\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\n\n \n \n \n \nSpam\nToday's meaning: Unsolicited electronic communications, for example by sending\nmass-emails or posting in forums and chats.\nThe term goes back to a sketch by the British comedy group Monty Python from 1970.\nIn the sketch, a cafe is including\nSpam (a brand of canned cooked\npork) in almost every dish. The excessive amount of Spam mentioned is a\nreference to the ubiquity of it and other imported canned meat products in the\nUK after World War II (a period of rationing in the UK) as the country struggled\nto rebuild its agricultural base.\nReference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Vintage Ad: Look What You Can Do With One\nCan of Spam \n \n \n \n \n \n Source: By user Jamie (jbcurio) on\nflickr.com\n \n \n \n\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\nMonty Pythons Flying Circus (1974) - SPAM from\nTesting Tester on Vimeo.\n\n \n \n \n \nRadio Button\nToday's meaning: A UI element that allows to choose from a predefined set of\nmutually exclusive options\n"Radio buttons" are named after the analogous pendant of mechanical buttons that\nwere used in radios. The UI concept has later been used in tape recorders,\ncassette recorders and wearable audio players (the famous "Walkman" and\nsimilar). And later in VCRs and video cameras.\nReference\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n An old car radio (left) and CSS\nradio buttons (right). Only a single option can be selected at any point in\ntime. As a kid, I would push two buttons at once so they would interlock. Good\ntimes.\n \n \n \n \n \n Source: Images by Matt Coady\n \n \n \n\n\n \n \n \n \nUppercase and lowercase\nToday's meaning: Distinction between capital letters and small letters on a\nkeyboard.\nBack when typesetting was a manual process where single letters made of led were\n"type set" to form words and sentences, upper- and lowercase letters were kept\nin separate containers β€” or cases β€” to make this rather tedious process a little faster.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n A set of printers cases\n \n \n \n \n \n Source: From the book 'Printing types, their history, forms, and use; a study in\nsurvivals' by Updike, Daniel Berkeley, 1860-1941. Freely available on\narchive.org.\n \n \n \n\n\n \n \n \n \nHonorable mentions\n\n \n \n \n \n404\nToday's meaning: HTTP Status Code for "File not found".\nThere is a story that the number comes from the server room where the World Wide\nWeb's central database was located. In there, administrators would manually\nlocate the requested files and transfer them, over the network, to the person\nwho made that request. If a file didn't exist, they'd return an error message:\n"Room 404: file not found".\nThis, however, seems to be a myth and the status code was chosen rather\narbitrarily based on the then well-established FTP status codes.\nReference\n\n \n \n \n \nProgramming languages and Abbreviations\nThe etymology of programming language names and common abbreviations would\nprobably warrant its own article, but I've decided to note down some of my\nfavorites for the time being.\n\n \n \n \n \nC++\nC++ is a programming language based on C by Bjarne Stroustrup. The name is a\nprogrammer pun by Rick Mascitti, a coworker of Stroustrup. The ++ refers to\nthe post-increment operator, that is common in many C-like languages. It\nincreases the value of a variable by 1. In that sense, C++ can be seen as the\nspiritual "successor" of C.\nReference\n\n \n \n \n \nC Sharp\nSimilarly to C++, C# is a C-like programming language. The name again refers to\n"incremental" improvements on top of C++. The # in the name looks like four\nplus signs. Hence C# == (C++)++. But on top of that, the name was also\ninspired by the musical notation where a sharp indicates that the written note\nshould be made a semitone higher in pitch.\nReference\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A C-Sharp note.\n \n \n \n \n \n Source: Wikimedia\nCommons\n \n \n \n\n\n \n \n \n \nPNG\nOfficially, PNG stands for Portable Network Graphics. It was born out of\nfrustration over a CompuServe announcement in 1994 that programs supporting GIF\nwould have to pay licensing fees from now on. A working group lead by hacker\nThomas Boutell created the .webp file format, a\npatent-free replacement for GIF. Therefore I prefer the format's unofficial\nname: PNG's Not GIF. Here's a great\narticle\non PNG's history.\nReference\n\n \n \n \n \nConclusion\n\nYou have to know the past to understand the present.\nβ€” Dr. Carl Sagan (1980)\n\nI hope you enjoyed this trip down memory lane. Now it's your turn!\nπŸ‘‰ Do you know any other stories? Send me a message, and I'll add them here.\n\n \n \n \n \nRelated Projects\n\nAwesome Computer History:\nA curated list of computer history videos, documentaries and related folklore\nmaintained by Thomas Watson.\nWikipedia: List of computer term etymologies:\nList of the origins of computer-related terms or terms used in the computing world.\nTalk: The Etymology of Programming by Brittany Storoz - JSConf EU\n2018: A talk that explains the background behind a few programming terms. Careful here: the explanation for "bug" is probably wrong as mentioned above.\nTypewriter terminology that has survived into the personal computer\nera:\nA list of computer terms that have their origins from typewriters.\nFolklore - The Original Macintosh:\nAnecdotes about the development of Apple's original Macintosh, and the people who made it.\n\n" }, { "title": "A Timelapse of Timelapse", "url": "https://endler.dev/2020/timelapse/", "body": "Timelapse is a little open-source screen\nrecorder for macOS. It takes a screenshot every second and creates a movie in\nthe end.\nTo celebrate its unlikely 1.0 release today, I present here a "timelapse" of\nthis project's journey. It just took ten years to get here.\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\n\n \n \n \n \n2011 - How it all began\nTo be honest, I don't remember why I initially wrote the tool. I must have had a\npersonal need for a screen recorder, I guess...\nIn May 2011, when I started the project, I was doing my Masters Degree in\nComputer Science. I might have needed the tool for University; most likely,\nhowever, I was just trying to find an excuse for not working on an assignment.\nDuring that time, I wrote a lot of tools like that. Mainly to scratch a personal\nitch, learn a new programming language, or just have fun.\nAmong them are tools like a random sandwich\ngenerator for Subway (the American\nfast-food chain), DrawRoom, a keyboard-driven\ndrawing app inspired by\nWriteRoom, and the\nobligatory CMS software, that I sold to\nclients. Surprisingly, none of them were a great success.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n DrawRoom, a tool that I wrote around the same time, is a real piece of art. To this day it has five commits and a single Github star (by myself, don't judge...).\n \n \n \n \n \n\nWhat I do know for sure is that I was unhappy with all existing screen\nrecorders. They could roughly be categorized into these three groups:\n\nProprietary solutions that cost money or could call home.\nTools that didn't work on macOS.\nSmall, fragile, one-off scripts that people passed around in forums or as\nGithub gists. They rarely worked as advertised.\n\nAmong the remaining tools were none that provided any timelapse functionality;\nso I set out to write my own.\nThis all sounds very epic, but in reality, I worked on it for a day. After five\nheroic commits on May 11, 2011, it sat there, idle, for seven years...\n\n \n \n \n \n2018\nA lot of time elapsed before anything exciting happened.\nIn January '18, seemingly out of nowhere, the first user filed a bug report. It\nwas titled hung when creating the\navi 😱. Turns out that a game\ndeveloper from Canada,\njuul1a, was trying\nto use the tool to track her progress on an indie game β€” how cool is that?\nTo help her out, I decided to do some general cleanup, finally write down some\ninstructions on how to even use the program, add a requirements.txt, and port\nthe tool from mencoder to\nffmpeg.\nAfter that, timelapse was ready for prime-time. 🎬 Here is some live action from\nher videos featuring timelapses:\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\nAt that point, the tool was still very wobbly and could only be used from the\ncommandline, but I began to see some potential for building a proper app from\nit; I just never found the time.\nIn October '18, I decided to ask for support during\nHacktoberfest. I created a few\ntickets and labeled them with hacktoberfest to try and find contributors.\nAnd then, I waited.\nFirst, Shreya V Prabhu fixed an issue where a\nnew recording was overwriting the previous one by adding a timestamp to the\nvideo name. Then Abner\nCampanha and Shane\nCreedon created a basic test structure.\nGbenro Selere added a CI pipeline for Travis CI.\nIt really worked, and the project was in much better shape after that!\n\n \n \n \n \n2019\nOne year passes by, and Kyle Jones adds some\ncontribution guidelines, while I move the CI pipeline to the newly released\nGithub actions.\nChaitanya fixed a bug where the program would hang when\nthe recording stopped by moving the video creation from threads to a separate\nprocess. He continued to make the\ncodebase more robust and became a core contributor, reviewing pull requests and\nhandling releases.\nThanks to orcutt989, the app now made use of\ntype hints in Python 3.6.\ngkpln3 added support for multi-monitor\nconfigurations. The screen captured will always be the one with the mouse on it.\n\n \n \n \n \n2020\nFast forward to today, and after almost ten years, we finally created a true\nmacOS app using the awesome py2app\nbundler. This should make the tool usable by non-developers.\n\n \n \n \n \nBack to the Future\nWe reached the end of our little journey.\nA long time has passed until 1.0. This project is a testament to the wonders of\nopen source collaboration, and I am proud to work on it with contributors from\naround the world. It doesn't have to be a life-changing project to bring people\ntogether who have fun building things. If this were the end of the story, I'd be\nokay with that. I doubt it, though. Here's to the next ten years!\n🎬 Download timelapse on Github.\n\n \n \n \n \nBonus\nThe video at the beginning is a timelapse of how I finish this article.\nHow meta.\n" }, { "title": "A Tiny, Static, Full-Text Search Engine using Rust and WebAssembly", "url": "https://endler.dev/2019/tinysearch/", "body": "\n I wrote a basic search module that you can add to a static website.\nIt's very lightweight (50kB-100kB gzipped) and works with Hugo, Zola, and\nJekyll. Only searching for entire words is supported. Try the search box on the\nleft for a demo. The code is on Github.\n\nStatic site generators are magical. They combine the best of both worlds:\ndynamic content without sacrificing performance.\nOver the years, this blog has been running on Jekyll, Cobalt, and, lately,\nZola.\nOne thing I always disliked, however, was the fact that static websites don't\ncome with "static" search engines, too. Instead, people resort to custom Google\nsearches, external search engines like Algolia, or pure\nJavaScript-based solutions like lunr.js or elasticlunr.\nAll of these work fine for most sites, but it never felt like the final answer.\nI didn't want to add yet another dependency on Google; neither did I want to use\na stand-alone web-backend like Algolia, which adds latency and is proprietary.\nOn the other side, I'm not a huge fan of JavaScript-heavy websites. For example,\njust the search indices that lunr creates can be multiple megabytes\nin size.\nThat feels lavish - even by today's bandwidth standards. On top of that,\nparsing JavaScript is still\ntime-consuming.\nI wanted some simple, lean, and self-contained search, that could be deployed\nnext to my other static content.\nAs a consequence, I refrained from adding search functionality to my blog at\nall. That's unfortunate because, with a growing number of articles, it gets\nharder and harder to find relevant content.\n\n \n \n \n \nThe Idea\nMany years ago, in 2013, I read "Writing a full-text search engine using Bloom\nfilters" β€” and it was a revelation.\nThe idea was simple: Let's run all articles through a generator that creates a\ntiny, self-contained search index using this magical data structure called a\n✨Bloom Filter ✨.\n\n \n \n \n \nWhat's a Bloom Filter?\nRoughly speaking, a Bloom filter is a space-efficient way to\ncheck if an element is in a set.\nThe trick is that it doesn't store the elements themselves; it just knows with\nsome confidence that they were stored before. In our case, it can say with a\ncertain error rate that a word is in an article.\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A Bloom filter stores a\n'fingerprint' (a number of hash values) of all input values instead of the raw\ninput. The result is a low-memory-footprint data structure. This is an example\nof 'hello' as an input.\n \n \n \n \n \n\nHere's the Python code from the original article that generates the Bloom\nfilters for each post (courtesy of Stavros\nKorokithakis):\n\nfilters = {}\nfor name, words in split_posts.items():\n filters[name] = BloomFilter(capacity=len(words), error_rate=0.1)\n for word in words:\n filters[name].add(word)\n\nThe memory footprint is extremely small, thanks to error_rate, which allows\nfor a negligible number of false positives.\nI immediately knew that I wanted something like this for my homepage. My idea\nwas to directly ship the Bloom filters and the search engine to the browser. I\ncould finally have a small, static search without the need for a backend!\n\n \n \n \n \nHeadaches\nThe disillusionment came quickly.\nI had no idea how to bundle and minimize the generated Bloom filters, let alone\nrun them on clients. The original article briefly touches on this:\n\nYou need to implement a Bloom filter algorithm on the client-side. This will\nprobably not be much longer than the inverted index search algorithm, but it’s\nstill probably a bit more complicated.\n\nI didn't feel confident enough in my JavaScript skills to pull this off. Back in\n2013, NPM was a mere three years old, and WebPack just turned one, so I also\ndidn't know where to look for existing solutions.\nUnsure what to do next, my idea remained a pipe dream.\n\n \n \n \n \nA New Hope\nFive years later, in 2018, the web had become a different place. Bundlers were\nubiquitous, and the Node ecosystem was flourishing. One thing, in particular,\nrevived my dreams about the tiny static search engine: webassembly.\n\nWebAssembly (abbreviated Wasm) is a binary instruction format for a\nstack-based virtual machine. Wasm is designed as a portable target for\ncompilation of high-level languages like C/C++/Rust, enabling deployment on\nthe web for client and server applications. [source]\n\nThis meant that I could use a language that I was familiar with to write the\nclient-side code β€” Rust! πŸŽ‰\nMy journey started with a prototype back in January\n2018.\nIt was just a direct port of the Python version from above:\n\nlet mut filters = HashMap::new();\nfor (name, words) in articles {\n let mut filter = BloomFilter::with_rate(0.1, words.len() as u32);\n for word in words {\n filter.insert(&word);\n }\n filters.insert(name, filter);\n}\n\nWhile I managed to create the Bloom filters for every article, I still had no\nclue how I should package that up for the web... until wasm-pack came along in\nFebruary\n2018.\n\n \n \n \n \nWhoops! I Shipped Some Rust Code To Your Browser.\nNow I had all the pieces of the puzzle:\n\nRust - A language I was comfortable with\nwasm-pack - A bundler for WebAssembly modules\nA working prototype that served as a proof-of-concept\n\nThe search box you see at the top of this page is the outcome. It fully runs on Rust using\nWebAssembly (a.k.a the RAW stack). Try it now if you like.\nThere were quite a few obstacles along the way.\n\n \n \n \n \nBloom Filter Crates\nI looked into a few Rust libraries (crates) that implement Bloom filters.\nFirst, I tried jedisct1's\nrust-bloom-filter, but the types\ndidn't implement\nSerialize/Deserialize.\nThis meant that I could not store my generated Bloom filters inside the binary and load\nthem from the client-side.\nAfter trying a few others, I found the\ncuckoofilter crate, which\nsupported serialization. The behavior is similar to Bloom filters, but if you're\ninterested in the differences, you can look at this\nsummary.\nHere's how to use it:\n\nlet mut cf = cuckoofilter::new();\n\n// Add data to the filter\nlet value: &str = "hello world";\nlet success = cf.add(value)?;\n\n// Lookup if data was added before\nlet success = cf.contains(value);\n// success ==> true\n\nLet's check the output size when bundling the filters for ten articles on my blog using cuckoo filters:\n\n~/C/p/tinysearch ❯❯❯ l storage\nPermissions Size User Date Modified Name\n.rw-r--r-- 44k mendler 24 Mar 15:42 storage\n\n44kB doesn't sound too shabby, but these are just the cuckoo filters for ten\narticles, serialized as a Rust binary. On top of that, we have to add the search\nfunctionality and the helper code. In total, the client-side code weighed in at\n216kB using vanilla wasm-pack. Too much.\n\n \n \n \n \nTrimming Binary Size\nAfter the sobering first result of 216kB for our initial prototype, we have a\nfew options to bring the binary size down.\nThe first is following johnthagen's advice on\nminimizing Rust binary size.\nBy setting a few options in our Cargo.toml, we can shave off quite a few bytes:\n\n"opt-level = 'z'" => 249665 bytes\n"lto = true" => 202516 bytes\n"opt-level = 's'" => 195950 bytes\n\nSetting opt-level to s means we trade size for speed,\nbut we're preliminarily interested in minimal size anyway. After all, a small download size also improves performance.\nNext, we can try wee_alloc, an alternative Rust allocator\nproducing a small .wasm code size.\n\nIt is geared towards code that makes a handful of initial dynamically sized allocations, and then performs its heavy lifting without any further allocations. This scenario requires some allocator to exist, but we are more than happy to trade allocation performance for small code size.\n\nExactly what we want. Let's try!\n\n"wee_alloc and nightly" => 187560 bytes\n\nWe shaved off another 4% from our binary.\nOut of curiosity, I tried to set codegen-units to 1, meaning we only use a single thread for code generation. Surprisingly, this resulted in a slightly smaller binary size.\n\n"codegen-units = 1" => 183294 bytes\n\nThen I got word of a Wasm optimizer called binaryen.\nOn macOS, it's available through homebrew:\n\nbrew install binaryen\n\nIt ships a binary called wasm-opt and that shaved off another 15%:\n\n"wasm-opt -Oz" => 154413 bytes\n\nThen I removed web-sys as we don't have to bind to the DOM: 152858 bytes.\nThere's a tool called twiggy to profile the code size of Wasm binaries.\nIt printed the following output:\n\ntwiggy top -n 20 pkg/tinysearch_bg.wasm\n Shallow Bytes β”‚ Shallow % β”‚ Item\n─────────────┼───────────┼────────────────────────────────\n 79256 β”Š 44.37% β”Š data[0]\n 13886 β”Š 7.77% β”Š "function names" subsection\n 7289 β”Š 4.08% β”Š data[1]\n 6888 β”Š 3.86% β”Š core::fmt::float::float_to_decimal_common_shortest::hdd201d50dffd0509\n 6080 β”Š 3.40% β”Š core::fmt::float::float_to_decimal_common_exact::hcb5f56a54ebe7361\n 5972 β”Š 3.34% β”Š std::sync::once::Once::call_once::{{closure}}::ha520deb2caa7e231\n 5869 β”Š 3.29% β”Š search\n\nFrom what I can tell, the biggest chunk of our binary is occupied by the raw data section for our articles.\nNext up, we got the function headers and some float to decimal helper functions, that most likely come from deserialization.\nFinally, I tried wasm-snip, which replaces a WebAssembly function's body with an unreachable like so, but it didn't reduce code size:\n\nwasm-snip --snip-rust-fmt-code --snip-rust-panicking-code -o pkg/tinysearch_bg_snip.wasm pkg/tinysearch_bg_opt.wasm\n\nAfter tweaking with the parameters of the cuckoo filters a bit and removing\nstop words from the articles, I\narrived at 121kB (51kB gzipped) β€” not bad considering the average image size on the web is around 900kB.\nOn top of that, the search functionality only gets loaded when a user clicks into the search field.\n\n \n \n \n \nFrontend and Glue Code\nwasm-pack will auto-generate the JavaScript code to talk to Wasm.\nFor the search UI, I customized a few JavaScript and CSS bits from\nw3schools.\nIt even has keyboard support!\nNow when a user enters a search query, we go through the cuckoo filter of each\narticle and try to match the words. The results are scored by the number of\nhits. Thanks to my dear colleague Jorge Luis Betancourt for adding that part.\n\n(Fun fact: this animation is about the same size as the uncompressed Wasm search itself.)\nOnly whole words are matched. I would love to add prefix-search, but the\nbinary became too big when I tried.\n\n \n \n \n \nUsage\nThe standalone binary to create the Wasm file is called tinysearch.\nIt expects a single path to a JSON file as an input:\n\ntinysearch path/to/corpus.json\n\nThis corpus.json contains the text you would like to index. The format is pretty straightforward:\n\n[\n {\n "title": "Article 1",\n "url": "https://example.com/article1",\n "body": "This is the body of article 1."\n },\n {\n "title": "Article 2",\n "url": "https://example.com/article2",\n "body": "This is the body of article 2."\n }\n]\n\nYou can generate this JSON file with any static site generator.\nHere's my version for Zola:\n\n{% set section = get_section(path="_index.md") %}\n\n[\n {%- for post in section.pages -%}\n {% if not post.draft %}\n {\n "title": {{ post.title | striptags | json_encode | safe }},\n "url": {{ post.permalink | json_encode | safe }},\n "body": {{ post.content | striptags | json_encode | safe }}\n }\n {% if not loop.last %},{% endif %}\n {% endif %}\n {%- endfor -%}\n]\n\nI'm pretty sure that the Jekyll version looks quite similar.\nHere's a starting point.\nIf you get something working for your static site generator, please let me know.\n\n \n \n \n \nObservations\n\nThis is still the wild west: unstable features, nightly Rust, documentation\ngets outdated almost every day.\nBring your thinking cap!\nCreating a product out of a good idea is a lot of work. One has to pay\nattention to many factors: ease-of-use, generality, maintainability,\ndocumentation, and so on.\nRust is very good at removing dead code, so you usually don't pay for what\nyou don't use. I would still advise you to be very conservative about the\ndependencies you add to a Wasm binary because it's tempting to add features\nthat you don't need and which will add to the binary size. For example, I\nused StructOpt during testing, and I had a main() function that was parsing\nthese command-line arguments. This was not necessary for Wasm, so I\nremoved it later.\nI understand that not everyone wants to write Rust code. It's complicated to\nget started with, but the cool thing is that you can\nuse almost any other language, too. For example, you can write Go code and\ntranspile to Wasm, or maybe you prefer PHP or Haskell. There is support for\nmany languages already.\nA lot of people dismiss WebAssembly as a toy technology. They couldn't be\nfurther from the truth. In my opinion, WebAssembly will revolutionize the way we build\nproducts for the web and beyond. What was very hard just two years ago is now\neasy: shipping code in any language to every browser. I'm super excited about\nits future.\nIf you're looking for a standalone, self-hosted search index for your company\nwebsite, check out sonic.\n\n\n ✨WOW! This tool getting quite a bit of traction lately.βœ¨β€\nI don't run ads on this website, but if you like these kind of experiments, please\nconsider sponsoring me on Github.\nThis allows me to write more tools like this in the future.\nAlso, if you're interested in hands-on Rust consulting, pick a date from my\ncalendar and we can talk about how I can help .\n\n\n \n \n \n \nTry it!\nThe code for tinysearch is on Github.\nPlease be aware of these limitations:\n\nOnly searches for entire words. There are no search suggestions.\nThe reason is that prefix search blows up binary size like Mentos and Diet Coke.\nSince we bundle all search indices for all articles into one static binary, I\nonly recommend to use it for low- to medium-size websites. Expect around 4kB\n(non-compressed) per article.\nThe compile times are abysmal at the moment (around 1.5 minutes after a\nfresh install on my machine), mainly because we're compiling the Rust crate\nfrom scratch every time we rebuild the index.\nUpdate: This is mostly fixed thanks to the awesome work of\nCephalonRho in PR\n#13. Thanks again!\n\nThe final Wasm code is laser-fast because we save the roundtrips to a\nsearch-server. The instant feedback loop feels more like filtering a list than\nsearching through files. It can even work fully offline, which might be nice if\nyou like to bundle it with an app.\n" }, { "title": "Maybe You Don't Need Kubernetes", "url": "https://endler.dev/2019/maybe-you-dont-need-kubernetes/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A woman riding a scooter\n \n \n \n \n \n Source: Illustration created by freepik, Nomad logo by HashiCorp.\n \n \n \n\nKubernetes is the 800-pound gorilla of container orchestration.\nIt powers some of the biggest deployments worldwide, but it comes\nwith a price tag.\nEspecially for smaller teams, it can be time-consuming to maintain and has a\nsteep learning curve. For what our team of four wanted to achieve at trivago, it\nadded too much overhead. So we looked into alternatives β€” and fell in love with\nNomad.\n\n \n \n \n \nThe wishlist\nOur team runs a number of typical services for monitoring and performance\nanalysis: API endpoints for metrics written in Go, Prometheus exporters, log\nparsers like Logstash or Gollum, and databases like InfluxDB or Elasticsearch.\nEach of these services run in their own container. We needed a simple system to\nkeep those jobs running.\nWe started with a list of requirements for container orchestration:\n\nRun a fleet of services across many machines.\nProvide an overview of running services.\nAllow for communication between services.\nRestart them automatically when they die.\nBe manageable by a small team.\n\nOn top of that, the following things were nice to have but not strictly\nrequired:\n\nTag machines by their capabilities (e.g., label machines with fast disks for\nI/O heavy services.)\nBe able to run these services independently of any orchestrator (e.g. in\ndevelopment).\nHave a common place to share configurations and secrets.\nProvide an endpoint for metrics and logging.\n\n\n \n \n \n \nWhy Kubernetes was not a good fit for us\nWhen creating a prototype with Kubernetes, we noticed that we started adding\never-more complex layers of logic to operate our services. Logic on which we\nimplicitly relied on.\nAs an example, Kubernetes allows embedding service configurations using\nConfigMaps. Especially when merging multiple config files or\nadding more services to a pod, this can get quite confusing quickly.\nKubernetes - or helm, for that matter - allows injecting external configs\ndynamically to ensure separation of concerns. But this can\nlead to tight, implicit coupling between your project and Kubernetes.\nHelm and ConfigMaps are optional features so you don’t have to use them. You\nmight as well just copy the config into the Docker image. However, it’s tempting\nto go down that path and build unnecessary abstractions that can later bite you.\nOn top of that, the Kubernetes ecosystem is still rapidly evolving. It takes a\nfair amount of time and energy to stay up-to-date with the best practices and\nlatest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list\ngoes on and on. Not all tools are necessary to get started with Kubernetes, but\nit’s hard to know which ones are, so you have to be at least aware of them.\nBecause of that, the learning curve is quite steep.\n\n \n \n \n \nWhen to use Kubernetes\nAt trivago specifically, many teams use Kubernetes and are quite happy with it.\nThese instances are managed by Google or Amazon however, which have the capacity to do so.\nKubernetes comes with amazing\nfeatures,\nthat make container orchestration at scale more manageable:\n\nFine-grained rights management\nCustom controllers allow getting logic into the cluster. These are just\nprograms that talk to the Kubernetes API.\nAutoscaling! Kubernetes can scale your services up and down on demand. It\nuses service metrics to do this without manual intervention.\n\nThe question is if you really need all those features. You can't rely on these\nabstractions to just work; you'll have to learn what's going on under the\nhood.\nEspecially in our team, which runs most services on-premise (because of its\nclose connection to trivago's core infrastructure), we didn't want to afford\nrunning our own Kubernetes cluster. We wanted to ship services instead.\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n\n \n \n \n \nBatteries not included\nNomad is the 20% of service orchestration that gets you 80% of the way. All it\ndoes is manage deployments. It takes care of your rollouts and restarts your\ncontainers in case of errors, and that's about it.\nThe entire point of Nomad is that it does less: it doesn’t include\nfine-grained rights management or advanced network policies, and that’s by\ndesign. Those components are provided as enterprise services, by a third-party,\nor not at all.\nI think Nomad hit a sweet-spot between ease of use and expressiveness. It's good\nfor small, mostly independent services. If you need more control, you'll have to\nbuild it yourself or use a different approach. Nomad is just an orchestrator.\nThe best part about Nomad is that it's easy to replace. There is little to no\nvendor lock-in because the functionality it provides can easily be integrated\ninto any other system that manages services. It just runs as a plain old single\nbinary on every machine in your cluster; that's it!\n\n \n \n \n \nThe Nomad ecosystem of loosely coupled components\nThe real power of Nomad lies within its ecosystem. It integrates very well with\nother - completely optional - products like Consul (a key-value store) or\nVault (for secrets handling). Inside your Nomad file, you can have sections\nfor fetching data from those services:\n\ntemplate {\n data = <<EOH\nLOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"\nAPI_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}"\nEOH\n\n destination = "secrets/file.env"\n env = true\n}\n\nThis will read the service/geo-api/log-verbosity key from Consul and expose it\nas a LOG_LEVEL environment variable inside your job. It's also exposing\nsecret/geo-api-key from Vault as API_KEY. Simple, but powerful!\nBecause it's so simple, Nomad can also be easily extended with other services\nthrough its API. For example, jobs can be tagged for service discovery. At\ntrivago, we tag all services, which expose metrics, with trv-metrics. This\nway, Prometheus finds the services via Consul and periodically scrapes the\n/metrics endpoint for new data. The same can be done for logs by integrating\nLoki for example.\nThere are many other examples for extensibility:\n\nTrigger a Jenkins job using a webhook and Consul watches to redeploy your\nNomad job on service config changes.\nUse Ceph to add a distributed file system to Nomad.\nUse fabio for load balancing.\n\nAll of this allowed us to grow our infrastructure organically without too much\nup-front commitment.\n\n \n \n \n \nFair warning\nNo system is perfect. I advise you not to use any fancy new features in\nproduction right now. There are bugs and missing features of course - but\nthat's also the case for\nKubernetes.\nCompared to Kubernetes, there is far less momentum behind Nomad. Kubernetes has\nseen around 75.000 commits and 2000 contributors so far, while Nomad sports about\n14.000 commits and 300 contributors. It will be hard for Nomad to keep up with\nthe velocity of Kubernetes, but maybe it doesn’t have to! The scope is much more\nnarrow and the smaller community could also mean that it'll be easier to get your\npull request accepted, in comparison to Kubernetes.\n\n \n \n \n \nSummary\nThe takeaway is: don't use Kubernetes just because everybody else does.\nCarefully evaluate your requirements and check which tool fits the bill.\nIf you're planning to deploy a fleet of homogenous services on large-scale\ninfrastructure, Kubernetes might be the way to go. Just be aware of the\nadditional complexity and operational costs. Some of these costs can be\navoided by using a managed Kubernetes environment like Google Kubernetes\nEngine or Amazon EKS.\nIf you're just looking for a reliable orchestrator that is easy to maintain and\nextendable, why not give Nomad a try? You might be surprised by how far it'll get you.\nIf Kubernetes were a car, Nomad would be a scooter. Sometimes you prefer one and\nsometimes the other. Both have their right to exist.\n" }, { "title": "What Is Rust Doing Behind the Curtains?", "url": "https://endler.dev/2018/cargo-inspect/", "body": "Rust allows for a lot of syntactic sugar, that makes it a pleasure to write. It is sometimes hard, however, to look behind the curtain and see what the compiler is really doing with our code.\n\nAt Rust Belt Rust 2018, I saw a talk by Tshepang Lekhonkhobe titled Syntax conveniences afforded by the compiler (Recording here).\nTo quote the abstract:\n\nThe Rust compiler provides a number of conveniences that make life easier for its users. It is good to know what these are, to avoid being mystified by what's going on under the hood... the less magical thinking we have of the world, the better.\n\nHe goes on to give a few examples of these conveniences:\n\nlifetime elisions\ntype inference\nsyntactic sugar\nimplicit dereferencing\ntype coercions\nhidden code (e.g. the prelude)\n\nIt was very educational and fun to see him compare code with and without these conveniences during the talk.\nComing home, I wanted to learn more. I wondered if there was a tool, which revealed what Rust was doing behind the curtains.\nOver on Reddit, I found a discussion about compiler flags to produce desugared output.\n(Note that I'm using rustup here to trigger the nightly compiler with the +nightly flag.)\n\nrustc +nightly -Zunpretty=hir example.rs\n\nHIR stands for high-level intermediate representation. This is basically an abstract syntax tree (AST) more suited for use by the compiler. It replaces syntactic sugar with basic building blocks that are easier to handle by the following compile steps.\nTo find out more, read this detailed write-up by Nico Matsakis.\n\nAnyway, the output looked surprisingly readable (see below). With some syntax highlighting and formatting, this could be quite a handy tool.\nI tried to use rustfmt on it, and it worked unreasonably well.\nMotivated by this quick win, I wrapped it up in a cargo subcommand and called it cargo-inspect.\nLet's try cargo-inspect on some real code!\n\n \n \n \n \nExample - Desugaring a range expression\nThe following examples can also be found in the project's examples folder.\nInput:\n\nfor n in 1..3 {\n println!("{}", n);\n}\n\nOutput of cargo-inspect:\n\nThat's the neatly formatted terminal output. It sports line numbers and colors, thanks to prettyprint, which is a library on top of bat. Maybe you can't read that, so here's the gist of it:\n\nmatch ::std::iter::IntoIterator::into_iter(\n ::std::ops::Range { start: 1, end: 3 })\n mut iter => loop {\n // ...\n },\n};\n\nWe can see that 1..3 gets converted into std::ops::Range { start: 1, end: 3 }.\nTo the compiler backend, these are absolutely the same. So this holds:\n\nassert_eq!((1..3), std::ops::Range { start: 1, end: 3 });\n\n \n \n \n \nExample - File handling\nInput:\n\nuse std::fs::File;\nuse std::io::Error;\n\nfn main() -> Result<(), Error> {\n let file = File::open("file.txt")?;\n Ok(())\n}\n\nOutput:\n\n#[prelude_import]\nuse std::prelude::v1::*;\n#[macro_use]\nextern crate std;\nuse std::fs::File;\nuse std::io::Error;\n\nfn main() -> Result<(), Error> {\n let file = match ::std::ops::Try::into_result(\n <File>::open("file.txt")) {\n ::std::result::Result::Err(err) =>\n #[allow(unreachable_code)]\n {\n #[allow(unreachable_code)]\n return ::std::ops::Try::from_error(\n ::std::convert::From::from(err))\n }\n ::std::result::Result::Ok(val) =>\n #[allow(unreachable_code)]\n {\n #[allow(unreachable_code)]\n val\n }\n };\n Ok(())\n}\n\nWe can see that the carrier operator ? gets desugared into a match on the Result of File::open. In case of an error, We apply std::convert::From::from to convert between error types. Otherwise, we simply return the Ok value.\n\n \n \n \n \nTalk\nOver at FOSDEM in Belgium, I was able to speak about the project in detail.\nHere is the recording:\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\n\n \n \n \n \nFuture work\nI'm not planning to rewrite the compiler here. rustc is doing a far greater job than I could. All this functionality already existed before; I'm merely trying to make the compiler more approachable for learners like me.\nRight now, the tool is quite fragile. It throws ugly error messages when things go wrong.\nIt mostly shines, when you run it on small, isolated example snippets.\n\n \n \n \n \nGet involved!\nOver on Github, I opened up a few issues for others to get involved.\nNamely, I wish there were options to:\n\n\nMake it work with cargo projects.\n\nShow the original code above the desugared code.\n\nShow only part of the full output\n\n...and much more.\n\nAlso, if you find a particularly exciting code example, don't be shy to contribute it to the examples folder.\n" }, { "title": "The Unreasonable Effectiveness of Excel Macros", "url": "https://endler.dev/2018/excel/", "body": "I never was a big fan of internships, partially because all the exciting\ncompanies were far away from my little village in Bavaria and partially because\nI was too shy to apply.\nOnly once I applied for an internship in Ireland as part of a school program.\nOur teacher assigned the jobs and so my friend got one at Apple and I ended up\nat a medium-sized IT distributor β€” let's call them PcGo.\n\nJudging by the website, the company looked quite impressive, but in reality, it\nwas just a secluded, grey warehouse in the rainy industrial area of Cork. Upon\narrival, I was introduced to my colleague Evgeny, who was the main (and only)\nemployee responsible for assembling desktop computers. From what I can tell, he\nran the shop. He just spoke broken English, so he handed me an electric\nscrewdriver and a box of screws, and I got to work. Together we assembled a lot\nof computers in my first week, and we had a lot of fun. One day he drove me home\nfrom work because I missed my bus. It was a rainy day and while he was driving\nthrough the narrow streets of Cork we talked and laughed, but all of a sudden I\nheard a loud bang. I looked through the rear mirror only to find that there was\nno rear mirror anymore. Turns out he bumped into another car, and the thing went\noff. Evgeny didn't mind. In a thick Eastern-European accent he remarked "Lost\nthree mirrors before already," and kept driving.\nIn my second week, I had a visit from my boss. Apparently, I was done with the\nworkload that they planned for my three-week internship. I was used to\nassembling and installing computers, which explains why.\nTo keep me busy, they put together another task. On an old Windows 98 computer\nin the back, he pointed the browser to silverpages.ie, searched for "computer"\nand after a while we looked at an endless list of addresses of Irish companies\nhaving "something to do with computers." Each entry consisted of the expected\nfields: the company name, the address, the phone number, the website (if any)\nand a list of keywords.\nMy boss said that they needed an overview of all competing vendors. He carefully\nselected a field from an entry, copied it and pasted it into an Excel sheet. He\ndid the same for the remaining fields. "That's it!", he said with a fake smile.\nWe both knew that this would mean two boring weeks for me.\nThey wanted to keep me busy by letting me manually scrape the entirety of a web database.\nI could have taken that as an insult, but instead, I looked at it as a\nchallenge.\nI noticed that the page number on silverpages.ie could be controlled by a GET\nparameter.\n"Can I write a program that does the scraping?" My boss was noticeably puzzled.\n"Uhm... you can do whatever you want, but you're not allowed to install any\nadditional software!". With that, he was off.\nJudging from the installed programs, I wasn't left with many choices: Excel or\nMinesweeper. I knew that Excel's Visual Basic macros were quite powerful, but I\nwasn't sure if I could scrape a full website with it.\nAfter a while, I detected a feature to download a website into an Excel sheet\n(what a glorious functionality). This worked perfectly, so all I had to do was\nrecord a macro to create a temporary sheet for each page, copy all important\nfields into a "master slide" and then get rid of the temporary sheet. I recorded\nthe macro and looked at the code. The rest of the day was spent figuring out how\nto modify the URL in a loop and cleaning up the macro. I pressed the "run macro"\nbutton and then I sat there waiting. The computer was running at full speed. My\nbiggest fear was that the program would crash or that the computer would run out\nof memory. I refrained from playing minesweeper on it, so I mostly played pool\nor chatted with Evgeny.\nWhen I came to the office the next morning, my program was done. To my surprise, it scraped the entirety\nof SilverPages, and there were many thousands of entries in the list. I sent the\ndocument to my boss via E-Mail and then got back to playing minesweeper.\nAn hour later, three guys with suits were standing behind me. I had to show them\nthe list again. They couldn't believe I did that on my own, so I showed them the\ntool to scrape the data. For them, I had some sort of superpower.\nThey left without giving me another task; I was free to do whatever I wanted for\nthe remaining two weeks. I went on to write an inventory tool for them, which\nthey could also manage via Excel. It was just a glorious Excel form for a\nspreadsheet that they maintained manually. I spent two weeks of my summer\nvacation to finish that tool because they said they would pay me for that, which, of course, they didn't :).\n\n \n \n \n \nLessons learned\n\nNever underestimate the power of Excel macros.\nIf you have a boring task at hand, make it more challenging by adding constraints.\n\n" }, { "title": "Switching from a German to a US Keyboard Layout - Is It Worth It?", "url": "https://endler.dev/2018/keyboard/", "body": "For the first three decades of my life, I've used a German keyboard layout.\nA few months ago, I switched to a US layout.\nThis post summarizes my thoughts around the topic.\nI was looking for a similar article before jumping the gun, but I couldn't find one β€” so I wrote it.\n\n \n \n \n \nWhy Switch?\nI was reasonably efficient when writing prose, but felt like\na lemur on a piano when programming:\nlots of finger-stretching while trying to the special keys, {, ;, or /.\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n German Keyboard Layout\n \n \n \n \n \n Source: Image by Wikipedia\n \n \n \n\nHere's Wikipedia's polite\nexplanation why the\nGerman keyboard sucks for programming:\n\nLike many other non-American keyboards, German keyboards change the right Alt\nkey into an Alt Gr key to access a third level of key assignments. This is\nnecessary because the umlauts and some other special characters leave no room\nto have all the special symbols of ASCII, needed by programmers among others,\navailable on the first or second (shifted) levels without unduly increasing\nthe size of the keyboard.\n\n\n \n \n \n \nBut Why Switch Now?\nAfter many years of using a rubber-dome Logitech Cordless Desktop\nWave, I\nhad to get a mechanical keyboard again.\nThose rubber domes just feel too mushy to me now. In addition to that, I enjoy the\nclicky sound of a mechanical keyboard and the noticeable tactile bump. (I'm using\nCherry MX Brown Keys with O-Ring dampeners to contain the anger of my coworkers.)\nMost mechanical keyboards come with an ANSI US layout only, so I figured, I'd\nfinally make the switch.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n My lovely keyboard β€” Durgod Taurus K320 (referral link). They also have a fancy white-pink ISO version now.\n \n \n \n \n \n\n\n \n \n \n \nHow Long Did It Take To Get Accustomed To The New Layout?\nWorking as a Software Engineer, my biggest fear was, that the switch would slow\ndown my daily work. This turned out not to be true. I was reasonably productive\nfrom day one, and nobody even noticed any difference. (That's a good thing,\nright?)\nAt first, I didn't like the bar-shaped US-Return key. I preferred the European\nlayout with a vertical enter key. I was afraid that I would hit the key by\naccident. After a while, I find that the US return key to be even more convenient.\nI never hit it by accident, and it's easy to reach with my pinky from the home position.\nWithin two weeks, I was back to 100% typing speed.\n\n \n \n \n \nDid My Programming Speed Improve Noticeably?\nYup. I'd say I can type programs about 30% faster now.\nEspecially when using special characters (/, ;, {, and so on) I'm much\nfaster now; partly because the key locations feel more intuitive, but mainly\nbecause my fingers stay at their dedicated positions now.\nSomehow the position of special characters feels just right. I can now understand the\nreason why Vim is using / for search or why the pipe symbol is |: both are\neasy to reach! It all makes sense now!\n(For a fun time, try that on a German keyboard!)\nI now understand why Mircosoft chose \\ as a directory separator: it's easily\naccessible from a US keyboard. On the German layout, it's… just… awful\n(Alt Gr+ß on Windows, Shift + Option + 7 on Mac).\nThe opening curly brace on a German layout Mac is produced with Alt+8, which\nalways made me leave the home\nrow and break my typing\nflow. Now there are dedicated keys for parentheses. Such a relief!\n\n \n \n \n \nAm I Slower When Writing German Texts Now?\nIn the beginning, I was.\nSomehow my brain associated the German layout with German\ntexts. First, I used the macOS layout switcher.\nThis turned out to be cumbersome and take time.\nThen I found the "US with Umlauts via Option Key\nLayout". It works perfectly fine for\nme. It allows me to use a single Keyboard layout but insert German umlauts at will\n(e.g. ΓΆ is Option+o). There is probably a similar layout for other language combinations.\n\n \n \n \n \nIs Switching Between Keyboards Painful?\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n US keyboard layout\n \n \n \n \n \n Source: Wikipedia\n \n \n \n\nMy built-in MacBook Pro keyboard layout is still German. I was afraid, that switching between\nthe internal German and the external English keyboard would confuse me. This\nturned out not to be a problem. I rarely look at the print anyway.\n(Update: can't remember when I last looked at the print.)\n\n \n \n \n \nSummary\nIf you consider switching, just do it. I don't look back at all.\n" }, { "title": "fastcat - A Faster `cat` Implementation Using Splice", "url": "https://endler.dev/2018/fastcat/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n\nLots of people asked me to write another piece about the internals of well-known\nUnix commands. Well, actually, nobody asked me, but it makes for a good\nintro. I'm sure you’ve read the previous parts about yes and\nls β€” they are epic.\nAnyway, today we talk about cat, which is used to concatenate files - or, more\ncommonly, abused to print a file's contents to the screen.\n\n# Concatenate files, the intended purpose\ncat input1.txt input2.txt input3.txt > output.txt\n\n# Print file to screen, the most common use case\ncat myfile\n\n \n \n \n \nImplementing cat\nHere's a naive cat in Ruby:\n\n#!/usr/bin/env ruby\n\ndef cat(args)\n args.each do |arg|\n IO.foreach(arg) do |line|\n puts line\n end\n end\nend\n\ncat(ARGV)\n\nThis program goes through each file and prints its contents line by line.\nEasy peasy! But wait, how fast is this tool?\nI quickly created a random 2 GB file for the benchmark.\nLet's compare the speed of our naive implementation with the system one\nusing the awesome pv (Pipe Viewer) tool.\nAll tests are averaged over five runs on a warm cache (file in memory).\n\n# Ruby 2.5.1\n> ./rubycat myfile | pv -r > /dev/null\n[196MiB/s]\n\nNot bad, I guess? How does it compare with my system's cat?\n\ncat myfile | pv -r > /dev/null\n[1.90GiB/s]\n\nUh oh, GNU cat is ten times faster than our little Ruby cat. 🐌\n\n \n \n \n \nMaking our Ruby cat a little faster\nOur naive Ruby code can be tweaked a bit.\nTurns out line buffering hurts performance in the end1:\n\n#!/usr/bin/env ruby\n\ndef cat(args)\n args.each do |arg|\n IO.copy_stream(arg, STDOUT)\n end\nend\n\ncat(ARGV)\n\nrubycat myfile | pv -r > /dev/null\n[1.81GiB/s]\n\nWow... we didn't really try hard, and we're already approaching the speed of a\ntool that gets optimized since\n1971. πŸŽ‰\nBut before we celebrate too much, let's see if we can go even faster.\n\n \n \n \n \nSplice\nWhat initially motivated me to write about cat was this comment by user\nwahern on\nHacker News:\n\nI'm surprised that neither GNU yes nor GNU cat uses splice(2).\n\nCould this splice thing make printing files even faster? β€” I was intrigued.\nSplice was first introduced to the Linux Kernel in 2006, and there is a nice\nsummary from Linus Torvalds himself,\nbut I prefer the description from the manpage:\n\nsplice() moves data between two file descriptors without copying\nbetween kernel address space and user address space. It transfers up\nto len bytes of data from the file descriptor fd_in to the file\ndescriptor fd_out, where one of the file descriptors must refer to a\npipe.\n\nIf you really want to dig deeper, here's the corresponding source code from the\nLinux Kernel,\nbut we don't need to know all the nitty-gritty details for now.\nInstead, we can just inspect the header from the C implementation:\n\n#include <fcntl.h>\n\nssize_t splice (int fd_in, loff_t *off_in, int fd_out,\n loff_t *off_out, size_t len,\n unsigned int flags);\n\nTo break it down even more, here's how we would copy the entire src file to dst:\n\nconst ssize_t r = splice (src, NULL, dst, NULL, size, 0);\n\nThe cool thing about this is that all of it happens inside the Linux kernel, which means we won't copy a single byte to userspace (where our program runs).\nIdeally, splice works by remapping pages and does not actually copy\nany data, which may improve I/O performance\n(reference).\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n Source: File icon by Aleksandr Vector from the Noun Project. Terminal icon by useiconic.com from the Noun Project.\n \n \n \n\n\n \n \n \n \nUsing splice from Rust\nI have to say I'm not a C programmer and I prefer Rust because it offers a safer\ninterface. Here's the same thing in Rust:\n\n#[cfg(any(target_os = "linux", target_os = "android"))]\npub fn splice(\n fd_in: RawFd,\n off_in: Option<&mut libc::loff_t>,\n fd_out: RawFd,\n off_out: Option<&mut libc::loff_t>,\n len: usize,\n flags: SpliceFFlags,\n) -> Result<usize>\n\nNow I didn't implement the Linux bindings myself. Instead, I just used a library called\nnix, which provides Rust friendly bindings to *nix APIs.\nThere is one caveat, though:\nWe cannot really copy the file directly to standard out, because splice\nrequires one file descriptor to be a pipe.\nThe way around that is to create a pipe, which consists of a reader and a\nwriter (rd and wr).\nWe pipe the file into the writer, and then we read from the pipe and push the data to stdout.\nYou can see that I use a relatively big buffer of 16384 bytes (214) to improve performance.\n\nextern crate nix;\n\nuse std::env;\nuse std::fs::File;\nuse std::io;\nuse std::os::unix::io::AsRawFd;\n\nuse nix::fcntl::{splice, SpliceFFlags};\nuse nix::unistd::pipe;\n\nconst BUF_SIZE: usize = 16384;\n\nfn main() {\n for path in env::args().skip(1) {\n let input = File::open(&path).expect(&format!("fcat: {}: No such file or directory", path));\n let (rd, wr) = pipe().unwrap();\n let stdout = io::stdout();\n let _handle = stdout.lock();\n\n loop {\n let res = splice(\n input.as_raw_fd(),\n None,\n wr,\n None,\n BUF_SIZE,\n SpliceFFlags::empty(),\n ).unwrap();\n\n if res == 0 {\n // We read 0 bytes from the input,\n // which means we're done copying.\n break;\n }\n\n let _res = splice(\n rd,\n None,\n stdout.as_raw_fd(),\n None,\n BUF_SIZE,\n SpliceFFlags::empty(),\n ).unwrap();\n }\n }\n}\n\nSo, how fast is this?\n\nfcat myfile | pv -r > /dev/null\n[5.90GiB/s]\n\nHoly guacamole. That's over three times as fast as system cat.\n\n \n \n \n \nOperating System support\n\nLinux and Android are fully supported.\nOpenBSD\nalso has some sort of splice implementation called\nsosplice. I haven't tested that, though.\nOn macOS, the closest thing to splice is its bigger brother,\nsendfile, which can send a\nfile to a socket within the Kernel. Unfortunately, it does not support sending\nfrom file to file.2 There's also\ncopyfile,\nwhich has a similar interface, but unfortunately, it is not zero-copy. (I\nthought so in the beginning, but I was\nwrong.)\nWindows doesn't provide zero-copy file-to-file transfer\n(only file-to-socket transfer using the TransmitFile API).\n\nNevertheless, in a production-grade\nimplementation, the splice support could be activated on systems that support\nit, while using a generic implementation as a fallback.\n\n \n \n \n \nNice, but why on earth would I want that?\nI have no idea. Probably you don't, because your bottleneck is somewhere else.\nThat said, many people use cat for piping data into another process like\n\n# Count all lines in C files\ncat *.c | wc -l\n\nor\n\ncat kittens.txt | grep "dog"\n\nIn this case, if you notice that cat is the bottleneck try fcat (but first,\ntry to avoid cat altogether).\nWith some more work, fcat could also be used to directly route packets from one\nnetwork card to another, similar to netcat.\n\n \n \n \n \nLessons learned\n\nThe closer we get to bare metal, the more our hard-won abstractions fall\napart, and we are back to low-level systems programming.\nApart from a fast cat, there's also a use-case for a slow cat: old computers.\nFor that purpose, there's... well.. slowcat.\n\nThat said, I still have no idea why GNU cat does not use splice on Linux. πŸ€”\nThe source code for fcat is on Github.\nContributions welcome!\n\n \n \n \n \nFootnotes\n1. Thanks to reader Freeky for making this code more idiomatic.↩\n2. Thanks to reader masklinn for the hint.↩\n" }, { "title": "That Octocat on the Wall", "url": "https://endler.dev/2018/github/", "body": "\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Photo of my office with Github's octocat on the wall over my couch\n \n \n \n \n \n\nSo I'm in a bit of a sentimental mood lately.\nGithub got acquired by Microsoft.\nWhile I think the acquisition was well-deserved, I still wish it didn't happen.\nLet me explain.\n\n \n \n \n \nMy early days\nI joined Github on 3rd of January 2010.\nSince I was a bit late to the game, my usual handle (mre) was already taken.\nSo I naively sent a mail to Github, asking if I could bag the name as it seemed to be abandoned.\nTo my surprise, I got an answer.\nThe response came from a guy named Chris Wanstrath.\nAll he wrote was "it's yours."\nThat was the moment I fell in love with Github.\nI felt encouraged to collaborate on projects, that everybody could contribute something valuable.\nOnly later I found out that Chris was one of the founders and the CEO of the company.\n\n \n \n \n \nLiving on Github\nBefore Github, there was SourceForge, and I only went there to download binaries.\nGithub showed me, that there was an entire community of like-minded people\nout there, who ❀️ to work on code in their free-time.\nTo me, Github is much more than a git interface; it's a social network.\nWhile other people browse Facebook or Instagram, I browse Github.\nI can still vividly remember getting my first star and my first issue on one of my projects coming from a real (!) person other than myself.\nAfter so many years, a pull-request still feels like the most personal gift anyone could give to me.\n\n \n \n \n \nGithub - the culture\nAfter a while, I started to admire some Github employees deeply:\n\nZach Holman (who is about my age) is a great writer, speaker, and one of the most creative developers I can think of.\nScott Chacon, who taught me a lot about git and whose presentation tool, showoff, I've used at University.\nTom Preston-Werner, who I admire for refusing an offer from Microsoft to pursue his dream and build Github, for establishing a super-nerdy company culture, and for Jekyll.\n\nAll three developers have since left the company.\nI can't help but notice that Github has changed.\nThe harassment accusations and letting Zach Holman go are only part of the story.\nIt has become a company like any other, maintaining a mature product.\nIt doesn't excite me anymore.\n\n \n \n \n \nAn alternative reality\nThere's still a bitter taste in my mouth when I think that Github has fallen prey to one of the tech giants. I loved Github while it was a small, friendly community of passionate developers.\nCould this have been sustainable?\nMaybe through paid features for project maintainers.\nYou see, if you do Open Source every day, it can be a lot of work.\nPeople start depending on your projects, and you feel responsible for keeping the lights on.\nTo ease the burden, I'd love to have deeper insights into my project usage: visitor statistics for longer than two weeks,\na front page where you could filter and search for events, a better way to handle discussions\n(which can get out of hand quickly), better CI integration Γ  la Gitlab.\nThese features would be targeted at the top 10% of Github users, a group of 3 million people.\nWould this be enough to pay the bills? Probably. Would it be enough to grow? Probably not.\n\n \n \n \n \nSo what?\nI don't think the acquisition will kill the culture. Microsoft is a strong partner and Nat Friedman is one of us.\nOn the other side, I'm not as enthusiastic as I used to be.\nThere's room for competitors now and I'm beginning to wonder what will be the next Github.\nThat said, I will keep the Octocat on my office wall, in the hope that the excitement comes back.\n" }, { "title": "Ten Years of Vim", "url": "https://endler.dev/2018/ten-years-of-vim/", "body": "\n\n\nWhen I opened Vim by accident for the first time, I thought it was broken. My\nkeystrokes changed the screen in unpredictable ways, and I wanted to undo things\nand quit. Needless to say, it was an unpleasant experience. There was something\nabout it though, that kept me coming back and it became my main editor.\nFast forward ten years (!) and I still use Vim.\nAfter all the Textmates and Atoms and PhpStorms I tried, I still find myself at home in Vim.\nPeople keep asking me: Why is that?\n\n \n \n \n \nWhy Vim?\nBefore Vim, I had used many other editors like notepad or nano. They all behaved more or less as expected: you insert text, you move your cursor with the arrow keys or your mouse, and you save with Control + S or by using the menu bar. VI (and Vim, its spiritual successor) is different.\nEVERYTHING in Vim is different, and that's why it's so highly effective. Let me explain.\n\n \n \n \n \nThe Zen of Vim\nThe philosophy behind Vim takes a while to sink in:\nWhile other editors focus on writing as the central part of working with text, Vim thinks it's editing.\nYou see, most of the time I don't spend writing new text; instead, I edit existing text.\nI mold text, form it, turn it upside down.\nWriting text is craftsmanship and hard work. You have to shape your thoughts with your cold, bare hands until they somewhat form a coherent whole.\nThis painful process is what Vim tries to make at least bearable. It helps you keep control.\nIt does that, by providing you sharp, effective tools to modify text.\nThe core of Vim is a language for editing text.\n\n \n \n \n \nVim, The Language\nThe Vim commands are not cryptic, you already know them.\n\nTo undo, type u.\nTo find the next t, type ft.\nTo delete a word, type daw.\nTo change a sentence, type cas.\n\nMore often than not, you can guess the correct command by thinking of an operation you want to execute and an object to execute it on.\nThen just take the first character of every word. Try it!\nIf anything goes wrong, you can always hit ESC and type u for undo.\nOperations: delete, find, change, back, insert, append,...\nObjects: word, sentence, parentheses, (html) tag,... (see :help text-objects)\nInserting text is just another editing operation, which can be triggered with i.\nThat's why, by default, you are in normal mode β€” also called command mode β€” where all those operations work.\nOnce you know this, Vim makes a lot more sense, and that's when you start to be productive.\n\n \n \n \n \nHow My Workflow Changed Over The Years\nWhen I was a beginner, I was very interested in how people with more Vim experience would use the editor.\nNow that I'm a long-time user, here's my answer: there's no secret sauce.\nI certainly feel less exhausted after editing text for a day, but 90% of the commands I use fit on a post-it note.\nThat said, throughout the years, my Vim habits changed.\nI went through several phases:\nYear 1: I'm happy if I can insert text and quit again.\nYear 2: That's cool, let's learn more shortcuts.\nYear 3-5: Let's add all the features!!!\nYear 6-10: My .vimrc is five lines long.\nYear three is when I started to learn the Vim ecosystem for real.\nI tried all sorts of flavors like MacVim and distributions like janus.\nFor a while, I even maintained my own Vim configuration\n, which was almost 400 lines long.\nAll of that certainly helped me learn what's out there, but I'm not sure if I would recommend that to a Vim beginner.\nAfter all, you don't really need all of that. Start with a vanilla Vim editor which works just fine!\nMy current Vim setup is pretty minimalistic. I don't use plugins anymore, mostly out of laziness and because built-in Vim commands or macros can replace them.\nHere are three concrete examples of how my workflow changed over the years:\n\n\nIn the beginning, I used a lot of "number powered movements". That is, if you have a command like b, which goes back one word in the text, you can also say 5b to go back five words. Nowadays I mostly use / to move to a matching word because it's quicker.\n\n\nI don't use arrow keys to move around in text anymore but forced myself to use h, j, k, l. Many people say that this is faster. After trying this for a few years, I don't think that is true (at least for me). I now just stick to it out of habit.\n\n\nOn my main working machine I use Vim for quick text editing and Visual Studio Code plus the awesome Vim plugin for projects. This way, I get the best of both worlds.\n\n\n\n \n \n \n \nWorkflow Issues I Still Struggle With\nAfter all these years I'm still not a Vim master β€” far from it.\nAs every other Vim user will tell you, we're all still learning.\nHere are a few things I wish I could do better:\n\nJumping around in longer texts: I know the basics, like searching (/), jumping to a matching bracket (%) or jumping to specific lines (for line 10, type 10G), but I still could use symbols more often for navigation.\nUsing visual mode for moving text around: Sometimes it can be quite complicated to type the right combination of letters to cut (delete) the text I want to move around. That's where visual mode (v) shines. It highlights the selected text. I should use it more often.\nMultiple registers for copy and paste: Right now I only use one register (like a pastebin) for copying text, but Vim supports multiple registers. That's cool if you want to move around more than one thing at the same time. Let's use more of those!\nTabs: I know how tabs work, but all the typing feels clunky. That's why I never extensively used them. Instead, I mostly use multiple terminal tabs or an IDE with Vim bindings for bigger projects.\n\n\n \n \n \n \nWould I learn Vim again?\nThat's a tough question to answer.\nOn one side, I would say no.\nThere's a steep learning curve in Vim and seeing all those modern IDEs become better at understanding the user's intent, editing text became way easier and faster in general.\nOn the other side, Vim is the fastest way for me to write down my thoughts and code. As a bonus, it runs on every machine and might well be around for decades to come. In contrast, I don't know if the IntelliJ shortcuts will be relevant in ten years (note: if you read this in the future and ask yourself "What is IntelliJ?", the answer might be no).\n\n \n \n \n \nTakeaways\nIf I can give you one tip, don't learn Vim by memorizing commands. Instead, look at your current workflow and try to make it better, then see how Vim can make that easier. It helps to look at other people using Vim to get inspired (Youtube link with sound).\nYou will spend a lot of time writing text, so it's well worth the time investment to learn one editor really well β€” especially if you are a programmer.\nAfter ten years, Vim is somehow ingrained in my mind. I think Vim when I'm editing text. It has become yet another natural language to me. I'm looking forward to the next ten years.\n" }, { "title": "Refactoring Go Code to Avoid File I/O in Unit Tests", "url": "https://endler.dev/2018/go-io-testing/", "body": "At work today, I refactored some simple Go code to make it more testable.\nThe idea was to avoid file handling in unit tests without mocking or using temporary files by separating data input/output and data manipulation.\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A gopher reading a long computer printout\n \n \n \n \n \n Source: Illustration by Marcus Olsson CC BY-NC-SA 4.0\n \n \n \n\nI was surprised that I couldn't find a simple explanation on sites like StackOverflow,\nwhich is why I wrote down some notes myself so that others can refer to it in the future.\n\n \n \n \n \nOur example code\nThe initial version looked like this:\n\npackage main\n\nimport (\n\t"bufio"\n\t"io/ioutil"\n\t"os"\n)\n\nfunc main() {\n\tanalyze("test.txt")\n}\n\nfunc analyze(file string) error {\n\thandle, err := os.Open(file)\n\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer handle.Close()\n\n\tscanner := bufio.NewScanner(handle)\n\tfor scanner.Scan() {\n\t\t// Do something with line\n\t\t_ = scanner.Text()\n\t}\n\treturn nil\n}\n\nAs you can see, we take a filename as input, and we open that file inside the analyze function to do something with its contents.\n\n \n \n \n \nWriting our first test for the code\nA typical test harness for that code might look like this:\n\npackage main\n\nimport "testing"\n\nfunc Test_analyze(t *testing.T) {\n\tt.Run("Test something", func(t *testing.T) {\n\t\tif err := analyze("test.txt"); (err != nil) != false {\n\t\t\tt.Errorf("analyze() error = %v", err)\n\t\t}\n\t})\n}\n\nAll fine and good?\n\n \n \n \n \nProblems\nThis will work, but file I/O while running tests is not always the best idea.\nFor one, you could be running in a constrained environment, where you don't have access to the file.\nWe could use temporary files to avoid this.\nBut there might be problems with disk I/O, which makes for flaky tests and frustration.\nAnother process could also modify the file during the test.\nAll these issues have nothing to do with your code.\nFurthermore, it's not enough to just look at the test and see exactly what's going on. You also have to read the text file first.\nA lot of people suggest mocking instead.\nThere are quite a few powerful libraries like spf13/afero for this purpose.\nThese packages will create temporary files in the background and clean up afterward.\nIn my opinion, mocking should be the last resort when it comes to testing. Before you mock, check that you use the right abstractions in your code.\nMaybe implementing against an interface or using Dependency Injection helps decouple components?\nMore often than not, a clear separation of concerns is all you need.\n\n \n \n \n \nRefactoring to make testing easier\nIn my case above, we can easily avoid using mocks and temporary files by decoupling file I/O from the analysis.\nWe do so by refactoring our analyze function to call doSomething, which takes an io.Reader.\n(You could also use an array of strings for now.)\nOur main.go now looks like this:\n\npackage main\n\nimport (\n\t"bufio"\n\t"io"\n\t"os"\n)\n\nfunc main() {\n\tanalyze("test.txt")\n}\n\nfunc analyze(file string) error {\n\thandle, err := os.Open(file)\n\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer handle.Close()\n\treturn doSomething(handle)\n}\n\nfunc doSomething(handle io.Reader) error {\n\tscanner := bufio.NewScanner(handle)\n\tfor scanner.Scan() {\n\t\t// Do something with line\n\t\t_ = scanner.Text()\n\t}\n\treturn nil\n}\n\nNow we can test the actual analysis in isolation:\n\npackage main\n\nimport (\n\t"strings"\n\t"testing"\n)\n\nfunc Test_analyze(t *testing.T) {\n\tt.Run("Test something", func(t *testing.T) {\n\t\tif err := doSomething(strings.NewReader("This is a test string")); (err != nil) != false {\n\t\t\tt.Errorf("analyze() error = %v", err)\n\t\t}\n\t})\n}\n\nWe changed analyze("test.txt") to doSomething(strings.NewReader("This is a test string")).\n(Of course, we should also write a separate test for analyze(), but the focus is on decoupling the datasource-agnostic part here.)\n\n \n \n \n \nResult\nBy slightly refactoring our code, we gained the following advantages:\n\nSimple testability: No mocks or temporary files.\nSeparation of concerns: Each function does exactly one thing.\nEasier code re-use: The doSomething() function will work with any io.Reader and can be called from other places. We can even move it to its own library if we want.\n\nOn Reddit, user soapysops made an important remark:\n\nIn general, I prefer to not accept a file name in an API. A file name doesn't give users enough control. It doesn't let you use an unusual encoding, special file permissions, or a bytes.Buffer instead of an actual file, for example. Accepting a file name adds a huge dependency to the code: the file system, along with all of its associated OS specific stuff.\nSo I probably would have eliminated the file name based API and only exposed one based on io.Reader. That way, you have complete code coverage, fast tests, and far fewer edge cases to worry about.\n\nI totally agree with that sentiment.\nBut often times you can't simply change the user-facing API easily, because the API might be public and might already have users.\nThe refactoring above is just the first step towards better architecture. There is definitely a lot more you can do.\nIf that got you interested, also check out justforfunc #29: dependency injection in a code review, which talks about the same topic.\n" }, { "title": "A Tiny `ls` Clone Written in Rust", "url": "https://endler.dev/2018/ls/", "body": "In my series of useless Unix tools rewritten in Rust, today I'm going to be covering one of my all-time favorites: ls.\nFirst off, let me say that you probably don't want to use this code as a replacement for ls on your local machine (although you could!).\nAs we will find out, ls is actually quite a powerful tool under the hood.\nI'm not going to come up with a full rewrite, but instead only cover the very basic output that you would expect from calling ls -l on your command line.\nWhat is this output? I'm glad you asked.\n\n \n \n \n \nExpected output\n\n> ls -l\ndrwxr-xr-x 2 mendler staff 13468 Feb 4 11:19 Top Secret\n-rwxr--r-- 1 mendler staff 6323935 Mar 8 21:56 Never Gonna Give You Up - Rick Astley.mp3\n-rw-r--r-- 1 mendler staff 0 Feb 18 23:55 Thoughts on Chess Boxing.doc\n-rw-r--r-- 1 mendler staff 380434 Dec 24 16:00 nobel-prize-speech.txt\n\nYour output may vary, but generally, there are a couple of notable things going on. From left to right, we've got the following fields:\n\nThe drwx things in the beginning are the file permissions (also called the file mode). If d is set, it's a directory. r means read, w means write and x execute.\nThis rwx pattern gets repeated three times for the current user, the group, and other computer users respectively.\nNext we got the hardlink count when referring to a file, or the number of contained directory entries when referring to a directory. (Reference)\nOwner name\nGroup name\nNumber of bytes in the file\nDate when the file was last modified\nFinally, the path name\n\nFor more in-depth information, I can recommend reading the manpage of ls from the GNU coreutils used in most Linux distributions and the one from Darwin (which powers MacOS).\nWhew, that's a lot of information for such a tiny tool.\nBut then again, it can't be so hard to port that to Rust, right? Let's get started!\n\n \n \n \n \nA very basic ls in Rust\nHere is the most bare-bones version of ls, which just prints all files in the current directory:\n\nuse std::fs;\nuse std::path::Path;\nuse std::error::Error;\nuse std::process;\n\nfn main() {\n\tif let Err(ref e) = run(Path::new(".")) {\n\t\tprintln!("{}", e);\n\t\tprocess::exit(1);\n\t}\n}\n\nfn run(dir: &Path) -> Result<(), Box<Error>> {\n\tif dir.is_dir() {\n\t\tfor entry in fs::read_dir(dir)? {\n\t\t\t\tlet entry = entry?;\n\t\t\t\tlet file_name = entry\n\t\t\t\t\t\t.file_name()\n\t\t\t\t\t\t.into_string()\n\t\t\t\t\t\t.or_else(|f| Err(format!("Invalid entry: {:?}", f)))?;\n\t\t\t\tprintln!("{}", file_name);\n\t\t}\n\t}\n\tOk(())\n}\n\n\nWe can copy that straight out of the documentation.\nWhen we run it, we get the expected output:\n\n> cargo run\nCargo.lock\nCargo.toml\nsrc\ntarget\n\nIt prints the files and exits. Simple enough.\nWe should stop for a moment and celebrate our success, knowing that we just wrote our first little Unix utility from scratch.\nPro Tip: You can install the binary with cargo install and call it like any other binary from now on.\nBut we have higher goals, so let's continue.\n\n \n \n \n \nAdding a parameter to specify the directory\nUsually, if we type ls mydir, we expect to get the file listing of no other directory than mydir. We should add the same functionality to our version.\nTo do this, we need to accept command line parameters.\nOne Rust crate that I love to use in this case is structopt. It makes argument parsing very easy.\nAdd it to your Cargo.toml. (You need cargo-edit for the following command).\n\ncargo add structopt\n\nNow we can import it and use it in our project:\n\n#[macro_use]\nextern crate structopt;\n\n// use std::...\nuse structopt::StructOpt;\n\n#[derive(StructOpt, Debug)]\nstruct Opt {\n\t/// Output file\n\t#[structopt(default_value = ".", parse(from_os_str))]\n\tpath: PathBuf,\n}\n\nfn main() {\n\tlet opt = Opt::from_args();\n\tif let Err(ref e) = run(&opt.path) {\n\t\t\tprintln!("{}", e);\n\t\t\tprocess::exit(1);\n\t}\n}\n\nfn run(dir: &PathBuf) -> Result<(), Box<Error>> {\n\t// Same as before\n}\n\nBy adding the Opt struct, we can define the command line flags, input parameters, and the help output super easily.\nThere are tons of configuration options, so it's worth checking out the project homepage.\nAlso note, that we changed the type of the path variable from Path to PathBuf. The difference is, that PathBuf owns the inner path string, while Path simply provides a reference to it. The relationship is similar to String and &str.\n\n \n \n \n \nReading the modification time\nNow let's deal with the metadata.\nFirst, we try to retrieve the modification time from the file.\nA quick look at the documentation shows us how to do it:\n\nuse std::fs;\n\nlet metadata = fs::metadata("foo.txt")?;\n\nif let Ok(time) = metadata.modified() {\n\tprintln!("{:?}", time);\n}\n\nThe output might not be what you expect: we receive a SystemTime object, which represents the measurement of the system clock. E.g. this code\n\nprintln!("{:?}", SystemTime::now());\n// Prints: SystemTime { tv_sec: 1520554933, tv_nsec: 610406401 }\n\nBut the format that we would like to have is something like this:\n\nMar 9 01:24\n\nThankfully, there is a library called chrono, which can read this format and convert it into any human readable output we like:\n\nlet current: DateTime<Local> = DateTime::from(SystemTime::now());\nprintln!("{}", current.format("%_d %b %H:%M").to_string());\n\nthis prints\n\n9 Mar 01:29\n\n(Yeah, I know it's getting late.)\nArmed with that knowledge, we can now read our file modification time.\n\ncargo add chrono\n\nuse chrono::{DateTime, Local};\n\nfn run(dir: &PathBuf) -> Result<(), Box<Error>> {\n\tif dir.is_dir() {\n\t\tfor entry in fs::read_dir(dir)? {\n\t\t\tlet entry = entry?;\n\t\t\tlet file_name = ...\n\n\t\t\tlet metadata = entry.metadata()?;\n\t\t\tlet size = metadata.len();\n\t\t\tlet modified: DateTime<Local> = DateTime::from(metadata.modified()?);\n\n\t\t\tprintln!(\n\t\t\t\t"{:>5} {} {}",\n\t\t\t\tsize,\n\t\t\t\tmodified.format("%_d %b %H:%M").to_string(),\n\t\t\t\tfile_name\n\t\t\t);\n\t\t}\n\t}\n\tOk(())\n}\n\nThis {:>5} might look weird. It's a formatting directive provided by std::fmt.\nIt means "right align this field with a space padding of 5" - just like our bigger brother ls -l is doing it.\nSimilarly, we retrieved the size in bytes with metadata.len().\n\n \n \n \n \nUnix file permissions are a zoo\nReading the file permissions is a bit more tricky.\nWhile the rwx notation is very common in Unix derivatives such as *BSD or GNU/Linux, many other operating systems ship their own permission management.\nThere are even differences between the Unix derivatives.\nWikipedia lists a few extensions to the file permissions that you might encounter:\n\n+ (plus) suffix indicates an access control list that can control additional permissions.\n. (dot) suffix indicates an SELinux context is present. Details may be listed with the command ls -Z.\n@ suffix indicates extended file attributes are present.\n\nThat just goes to show, that there are a lot of important details to be considered when implementing this in real life.\n\n \n \n \n \nImplementing very basic file mode\nFor now, we just stick to the basics and assume we are on a platform that supports the rwx file mode.\nBehind the r, the w and the x are in reality octal numbers. That's easier for computers to work with and many hardcore users even prefer to type the numbers over the symbols.\nThe ruleset behind those octals is as follows. I took that from the chmod manpage.\n\n\tModes may be absolute or symbolic.\n\tAn absolute mode is an octal number constructed\n\tfrom the sum of one or more of the following values\n\n\t 0400 Allow read by owner.\n\t 0200 Allow write by owner.\n\t 0100 For files, allow execution by owner.\n\t 0040 Allow read by group members.\n\t 0020 Allow write by group members.\n\t 0010 For files, allow execution by group members.\n\t 0004 Allow read by others.\n\t 0002 Allow write by others.\n\t 0001 For files, allow execution by others.\n\nFor example, to set the permissions for a file so that the owner can read, write and execute it and nobody else can do anything would be 700 (400 + 200 +100).\nGranted, those numbers are the same since the 70s and are not going to change soon, but it's still a bad idea to compare our file permissions directly with the values; if not for compatibility reasons, then for readability and to avoid magic numbers in our code.\nTherefore, we use the libc crate, which provides constants for those magic numbers.\nAs mentioned above, these file permissions are Unix specific, so we need to import a Unix-only library named std::os::unix::fs::PermissionsExt; for that.\n\nextern crate libc;\n\n// Examples:\n// * `S_IRGRP` stands for "read permission for group",\n// * `S_IXUSR` stands for "execution permission for user"\nuse libc::{S_IRGRP, S_IROTH, S_IRUSR, S_IWGRP, S_IWOTH, S_IWUSR, S_IXGRP, S_IXOTH, S_IXUSR};\nuse std::os::unix::fs::PermissionsExt;\n\nWe can now get the file permissions like so:\n\nlet metadata = entry.metadata()?;\nlet mode = metadata.permissions().mode();\nparse_permissions(mode as u16);\n\nparse_permissions() is a little helper function defined as follows:\n\nfn parse_permissions(mode: u16) -> String {\n\tlet user = triplet(mode, S_IRUSR, S_IWUSR, S_IXUSR);\n\tlet group = triplet(mode, S_IRGRP, S_IWGRP, S_IXGRP);\n\tlet other = triplet(mode, S_IROTH, S_IWOTH, S_IXOTH);\n\t[user, group, other].join("")\n}\n\nIt takes the file mode as a u16 (simply because the libc constants are u16)\nand calls triplet on it.\nFor each flag read, write, and execute, it runs a binary & operation on mode.\nThe output is matched exhaustively against all possible permission patterns.\n\nfn triplet(mode: u16, read: u16, write: u16, execute: u16) -> String {\n\tmatch (mode & read, mode & write, mode & execute) {\n\t\t(0, 0, 0) => "---",\n\t\t(_, 0, 0) => "r--",\n\t\t(0, _, 0) => "-w-",\n\t\t(0, 0, _) => "--x",\n\t\t(_, 0, _) => "r-x",\n\t\t(_, _, 0) => "rw-",\n\t\t(0, _, _) => "-wx",\n\t\t(_, _, _) => "rwx",\n\t}.to_string()\n}\n\n \n \n \n \nWrapping up\nThe final output looks like this. Close enough.\n\n> cargo run\nrw-r--r-- 7 6 Mar 23:10 .gitignore\nrw-r--r-- 15618 8 Mar 00:41 Cargo.lock\nrw-r--r-- 185 8 Mar 00:41 Cargo.toml\nrwxr-xr-x 102 5 Mar 21:31 src\nrwxr-xr-x 136 6 Mar 23:07 target\n\nThat's it! You can find the final version of our toy ls on Github.\nWe are still far away from a full-fledged ls replacement, but at least we learned a thing or two about its internals.\nIf you're looking for a proper ls replacement written in Rust, go check out lsd.\nIf, instead, you want to read another blog post from the same series, check out A Little Story About the yes Unix Command.\n" }, { "title": "Rust in 2018", "url": "https://endler.dev/2018/rust-2018/", "body": "I wrote about the future of Rust before and it seems like nobody stops me from doing it again! Quite the contrary: this time the Rust core team even asked for it.\nI'm a bit late to the party, but here are my 2 cents about the priorities for Rust in 2018.\n\n \n \n \n \nWho is this guy?\nThere's a depressingly high chance that we've never met before β€” which is a real shame.\nFor some context: I come from dynamically typed languages like Python and PHP.\nRust was the first language that allowed me to write real low-level code without feeling like arguing with a bouncer.\nTo me, Rust is not a fireflower, it's my own personal Megazord1.\nI want Rust to win, but for that, we need to tick a few points off the list.\n\n \n \n \n \nCompiler documentation for easier contribution\n\nWhen I was in Columbus, Ohio for Rust Belt Rust, I met Niko Matsakis, Ariel Ben-Yehuda, and Santiago Pastorino.\nThose fine gentlemen eagerly worked on non-lexical lifetimes during the impl-period.\nWatching them hack away on the compiler was deeply inspirational to me, and I started wondering if I could contribute, too.\nNeedless to say, the barrier to entry for hacking on the compiler can be quite high.\nI didn't contribute anything yet.\nOne thing I'd love to do is to spend short 30-60 minute chunks of time to fix a small thing in the compiler here and there. Could be as simple as renaming a variable, writing a test or adding some documentation.\nHence my first wish is, that contributing to the language will become easier.\nThat could be achieved by providing extensive mentorship, more entry-level tickets, and better compiler documentation.\nAll of that was already suggested by Niko.\n\n \n \n \n \nMore resources for intermediate programmers\nOn a related note, I'd like to see more talks/guidelines/books targeting intermediate Rust programmers.\nThis includes discussions on how to structure big projects in Rust and Rust-specific design patterns.\nI want to read more about professional Rust usage and see case-studies from various industries.\nFor example, there is a startup called snips, which builds an on-device voice-assistant using Rust.\nThey integrate with C and C++ libraries and I want to hear more about their journey.\n\n \n \n \n \nImprove the RFC process\nI try to follow the RFC process very closely, but my time is limited.\nMy wish is, that I can open any RFC and immediately get its status:\n\nA summary of the discussion with major pros and cons.\nA simple usage example, right at the beginning.\nThe next steps towards stabilization.\n\nFor example, if I look at this (not so) random issue, I don't even know where to start. What are the biggest blockers right now? Who is actively pushing this forward? How can I help out?\nGithub is great for code, but conversations about new features regularly get out of hand.\nThis is not a problem, that is limited to Rust, either. Just look at other big projects like Docker, Kubernetes, or Node.\nMaybe we need a new tool for that.\n\n \n \n \n \nThe usual suspects\nIf I could ask for two stable features in 2018, it would be ? in main\nand non-lexical lifetimes.\nThere's more I could mention of course, but I'm not gonna bore you with faster compile times, impl trait, generators, and the like.\nWe're on a good way here, see Nick Cameron's post instead.\nI'm convinced, that by improving documentation and mentorship, we can grow the number of contributors significantly\nand stabilize many highly-anticipated features this year.\n1. Disclaimer: I never watched a single episode of Power Rangers.↩\n" }, { "title": "Functional Programming for Mathematical Computing", "url": "https://endler.dev/2018/functional-mathematics/", "body": "Programming languages help us describe general solutions for problems; the result just happens to be executable by machines. Every programming language comes with a different set of strengths and weaknesses, one reason being that its syntax and semantics heavily influence the range of problems which can easily be tackled with it.\ntl;dr: I think that functional programming is better suited for mathematical computations than the more common imperative approach.\n\n \n \n \n \nUsing built-in abstractions for Mathematics\nThe ideas behind a language (the underlying programming paradigms) are distinctive for the community that builds around it. The developers create a unique ecosystem of ready-to-use libraries and frameworks around the language core. As a consequence, some languages are stronger in areas such as business applications (one could think of Cobol), others work great for systems programming (like C or Rust).\nWhen it comes to solving mathematical and numerical problems with computers, Fortran might come to mind. Although Fortran is a general-purpose language, it is mostly known for scientific computing. Of course, the language was created with that purpose in mind – hence the name, Formula Translation.\nOne reason for its popularity in this area is that it offers some built-in domain-specific keywords to express mathematical concepts, while keeping an eye on performance. For instance, it has a dedicated datatype for complex numbers – COMPLEX – and a keyword named DIMENSION which is quite similar to the mathematical term and can be used to create arrays and vectors.\n\n \n \n \n \nImperative vs functional style\nBuilt-in keywords can help expand the expressiveness of a language into a specific problem space, but this approach is severly limited. It’s not feasible to extend the language core ad infinitum; it would just be harder to maintain and take longer to learn. Therefore, most languages provide other ways of abstraction – like functions, subroutines, classes and objects – to split a routine into smaller, more manageable parts. These mechanisms might help to control the complexity of a program, but especially when dealing with mathematical problems, one has to be careful not to obfuscate the solution with boilerplate code.\n\n \n \n \n \nSpecimen I - Factorial\nAs an example, the stated problem might be to translate the following formula, which calculates the factorial of a positive number n, into program code:\n\nAn implementation of the above formula using imperative style Java might look like this:\n\npublic static long fact(final int n) {\n if (n < 0) {\n // Negative numbers not allowed\n return 0;\n }\n long prod = 1;\n for (int i = 1; i <= n; ++i) {\n prod *= i;\n }\n return prod;\n}\n\nThis is quite a long solution for such a short problem definition.\n(Note that writing a version with an explicit loop from 1 to n was on purpose; a recursive function would be shorter, but uses a concept which was not introduced by the mathematical formula.)\nAlso, the program contains many language-specific keywords, such as public, static, and System.err.println(). On top of that, the programmer must explicitly provide all data types for the variables in use – a tiresome obligation.\nAll of this obfuscates the mathematical definition.\nCompare this with the following version written in a functional language, like Haskell.\n\nfact n = product [1..n]\n\nThis is an almost direct translation from the problem definition into code. It needs no explicit types, no temporary variables and no access modifiers (such as public).\n\n \n \n \n \nSpecimen II - Dot product\nOne could argue that the above Haskell program owes its brevity to the fact, that the language provides just the right abstractions (namely the product keyword and the [1..n] range syntax) for that specific task.\nTherfore let’s examine a simple function which is neither available in Haskell nor in Java: The dot product of two vectors. The mathematical definition is as follows:\n \nFor vectors with three dimensions, it can be written as\n\nFirst, a Haskell implementation:\n\ntype Scalar a = a\ndata Vector a = Vector a a a deriving (Show)\ndot :: (Num a) => Vector a -> Vector a -> Scalar a\n(Vector a1 a2 a3) `dot` (Vector b1 b2 b3) = a1*b1 + a2*b2 + a3*b3\n\nNote, that the mathematical types can be defined in one line each. Further note, that we define the dot function in infix notation, that is, we place the first argument of dot in front of the function name and the second argument behind it. This way, the code looks more like its mathematical equivalent.\nAn example call of the above function would be \n\n(Vector 1 2 3) ’dot’ (Vector 3 2 1)\n\nwhich is short, precise and readable.\nNow, a similar implementation in Java.\n\npublic static class Vector<T extends Number> {\n private T x, y, z;\n\n public Vector(T x, T y, T z) {\n this.x = x;\n this.y = y;\n this.z = z;\n }\n\n public double dot(Vector<?> v) {\n return (x.doubleValue() * v.x.doubleValue() +\n y.doubleValue() * v.y.doubleValue() +\n z.doubleValue() * v.z.doubleValue());\n }\n }\n\n public static void main(String[] args) {\n Vector<Integer> a = new Vector<Integer>(3, 2, 1);\n Vector<Integer> b = new Vector<Integer>(1, 2, 3);\n System.out.println(a.dot(b));\n }\n}\n\nFor a proper textual representation of Vectors, the toString() Method would also need to be overwritten. In Haskell, one can simply derive from the Show typeclass as shown in the code.\n\n \n \n \n \nCreating new abstractions\nIf functions and types are not sufficient to write straightforward programs, Haskell also offers simple constructs to create new operators and keywords which extend the language core itself. This makes domain-specific-languages feasible and enables the developer to work more directly on the actual problem instead of working around peculiarities of the programming language itself (such as memory management or array iteration). Haskell embraces this concept; Java has no such functionality.\n\n \n \n \n \nConclusion\nI'm not trying to bash Java or worship Haskell here. Both languages have their place.\nI merely picked Java, because lots of programmers can read it.\nThe comparison is more between a functional and an imperative approach for numerical and symbolical programming; and for that, I prefer a functional approach every day. It removes clutter and yields elegant solutions. It provides convenient methods to work on a high level of abstraction and speak in mathematical terms and still, these strengths are disregarded by many programmers.\nAbraham H. Maslow’s observation in his 1966 book The Psychology of Science seems fitting:\n\nβ€œI suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”\n\n" }, { "title": "Rust for Rubyists", "url": "https://endler.dev/2017/rust-for-rubyists/", "body": "Recently I came across a delightful article on idiomatic Ruby.\nI'm not a good Ruby developer by any means, but I realized, that a lot of the patterns are also quite common in Rust.\nWhat follows is a side-by-side comparison of idiomatic code in both languages.\nThe Ruby code samples are from the original article.\n\n \n \n \n \nMap and Higher-Order Functions\nThe first example is a pretty basic iteration over elements of a container using map.\n\n\n\nuser_ids = users.map { |user| user.id }\n\nThe map concept is also pretty standard in Rust.\nCompared to Ruby, we need to be a little more explicit here:\nIf users is a vector of User objects, we first need to create an iterator from it:\n\n\n\nlet user_ids = users.iter().map(|user| user.id);\n\nYou might say that's quite verbose, but this additional abstraction allows us to express an important concept:\nwill the iterator take ownership of the vector, or will it not?\n\nWith iter(), you get a "read-only view" into the vector. After the iteration, it will be unchanged.\nWith into_iter(), you take ownership over the vector. After the iteration, the vector will be gone.\nIn Rust terminology, it will have moved.\nRead some more about the difference between iter() and into_iter() here.\n\nThe above Ruby code can be simplified like this:\n\n\n\nuser_ids = users.map(&:id)\n\nIn Ruby, higher-order functions (like map) take blocks or procs as an argument and the language provides a convenient shortcut for method invocation β€” &:id is the same as {|o| o.id()}.\nSomething similar could be done in Rust:\n\n\n\nlet id = |u: &User| u.id;\nlet user_ids = users.iter().map(id);\n\nThis is probably not the most idiomatic way to do it, though. What you will see more often is the use of Universal Function Call Syntax in this case:1\n\n\n\nlet user_ids = users.iter().map(User::id);\n\nIn Rust, higher-order functions take functions as an argument. Therefore users.iter().map(Users::id) is more or less equivalent to users.iter().map(|u| u.id()).2\nAlso, map() in Rust returns another iterator and not a collection.\nIf you want a collection, you would have to run collect() on that, as we'll see later.\n\n \n \n \n \nIteration with Each\nSpeaking of iteration, one pattern that I see a lot in Ruby code is this:\n\n\n\n["Ruby", "Rust", "Python", "Cobol"].each do |lang|\n puts "Hello #{lang}!"\nend\n\nSince Rust 1.21, this is now also possible:\n\n\n\n["Ruby", "Rust", "Python", "Cobol"]\n .iter()\n .for_each(|lang| println!("Hello {lang}!", lang = lang));\n\nAlthough, more commonly one would write that as a normal for-loop in Rust:\n\n\n\nfor lang in ["Ruby", "Rust", "Python", "Cobol"].iter() {\n println!("Hello {lang}!", lang = lang);\n}\n\n\n \n \n \n \nSelect and filter\nLet's say you want to extract only even numbers from a collection in Ruby.\n\n\n\neven_numbers = [1, 2, 3, 4, 5].map { |element| element if element.even? } # [ni, 2, nil, 4, nil]\neven_numbers = even_numbers.compact # [2, 4]\n\nIn this example, before calling compact, our even_numbers array had nil entries.\nWell, in Rust there is no concept of nil or Null. You don't need a compact.\nAlso, map doesn't take predicates. You would use filter for that:\n\n\n\nlet even_numbers = vec![1, 2, 3, 4, 5]\n .iter()\n .filter(|&element| element % 2 == 0);\n\nor, to make a vector out of the result\n\n\n\n// Result: [2, 4]\nlet even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]\n .into_iter()\n .filter(|element| element % 2 == 0).collect();\n\nSome hints:\n\nI'm using the type hint Vec<i64> here because, without it, Rust does not know what collection I want to build when calling collect.\nvec! is a macro for creating a vector.\nInstead of iter, I use into_iter. This way, I take ownership of the elements in the vector. With iter() I would get a Vec<&i64> instead.\n\nIn Rust, there is no even method on numbers, but that doesn't keep us from defining one!\n\n\n\nlet even = |x: &i64| x % 2 == 0;\nlet even_numbers = vec![1, 2, 3, 4, 5].into_iter().filter(even);\n\nIn a real-world scenario, you would probably use a third-party package (crate) like num for numerical mathematics:\n\n\n\nextern crate num;\nuse num::Integer;\n\nfn main() {\n let even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]\n .into_iter()\n .filter(|x| x.is_even()).collect();\n}\n\nIn general, it's quite common to use crates in Rust for functionality that is not in the standard lib.\nPart of the reason why this is so well accepted is that cargo is such a rad package manager.\n(Maybe because it was built by no other than Yehuda Katz of Ruby fame. πŸ˜‰)\nAs mentioned before, Rust does not have nil. However, there is still the concept of operations that can fail.\nThe canonical type to express that is called Result.\nLet's say you want to convert a vector of strings to integers.\n\n\n\nlet maybe_numbers = vec!["1", "2", "nah", "nope", "3"];\nlet numbers: Vec<_> = maybe_numbers\n .into_iter()\n .map(|i| i.parse::<u64>())\n .collect();\n\nThat looks nice, but maybe the output is a little unexpected. numbers will also contain the parsing errors:\n\n\n\n[Ok(1), Ok(2), Err(ParseIntError { kind: InvalidDigit }), Err(ParseIntError { kind: InvalidDigit }), Ok(3)]\n\nSometimes you're just interested in the successful operations.\nAn easy way to filter out the errors is to use filter_map:\n\n\n\nlet maybe_numbers = vec!["1", "2", "nah", "nope", "3"];\nlet numbers: Vec<_> = maybe_numbers\n .into_iter()\n .filter_map(|i| i.parse::<u64>().ok())\n .collect();\n\nI changed two things here:\n\nInstead of map, I'm now using filter_map.\nparse returns a Result, but filter_map expects an Option. We can convert a Result into an Option by calling ok() on it3.\n\nThe return value contains all successfully converted strings:\n\n\n\n[1, 2, 3]\n\nThe filter_map is similar to the select method in Ruby:\n\n\n\n[1, 2, 3, 4, 5].select { |element| element.even? }\n\n\n \n \n \n \nRandom numbers\nHere's how to get a random number from an array in Ruby:\n\n\n\n[1, 2, 3].sample\n\nThat's quite nice and idiomatic!\nCompare that to Rust:\n\n\n\nlet mut rng = thread_rng();\nrng.choose(&[1, 2, 3, 4, 5])\n\nFor the code to work, you need the rand crate. Click on the snippet for a running example.\nThere are some differences to Ruby. Namely, we need to be more explicit about what random number generator\nwe want exactly. We decide for a lazily-initialized thread-local random number generator, seeded by the system.\nIn this case, I'm using a slice instead of a vector. The main difference is that the slice has a fixed size while the vector does not.\nWithin the standard library, Rust doesn't have a sample or choose method on the slice itself.\nThat's a design decision: the core of the language is kept small to allow evolving the language in the future.\nThis doesn't mean that you cannot have a nicer implementation today.\nFor instance, you could define a Choose trait and implement it for [T].\n\n\n\nextern crate rand;\nuse rand::{thread_rng, Rng};\n\ntrait Choose<T> {\n fn choose(&self) -> Option<&T>;\n}\n\nimpl<T> Choose<T> for [T] {\n fn choose(&self) -> Option<&T> {\n let mut rng = thread_rng();\n rng.choose(&self)\n }\n}\n\nThis boilerplate could be put into a crate to make it reusable for others.\nWith that, we arrive at a solution that rivals Ruby's elegance.\n\n\n\n[1, 2, 4, 8, 16, 32].choose()\n\n\n \n \n \n \nImplicit returns and expressions\nRuby methods automatically return the result of the last statement.\n\n\n\ndef get_user_ids(users)\n users.map(&:id)\nend\n\nSame for Rust. Note the missing semicolon.\n\n\n\nfn get_user_ids(users: &[User]) -> Vec<u64> {\n users.iter().map(|user| user.id).collect()\n}\n\nBut in Rust, this is just the beginning, because everything is an expression.\nThe following block splits a string into characters, removes the h, and returns the result as a HashSet.\nThis HashSet will be assigned to x.\n\n\n\nlet x: HashSet<_> = {\n // Get unique chars of a word {'h', 'e', 'l', 'o'}\n let unique = "hello".chars();\n // filter out the 'h'\n unique.filter(|&char| char != 'h').collect()\n};\n\nSame works for conditions:\n\n\n\nlet x = if 1 > 0 { "absolutely!" } else { "no seriously" };\n\nSince a match statement is also an expression, you can assign the result to a variable, too!\n\n\n\nenum Unit {\n Meter,\n Yard,\n Angstroem,\n Lightyear,\n}\n\nlet length_in_meters = match unit {\n Unit::Meter => 1.0,\n Unit::Yard => 0.91,\n Unit::Angstroem => 0.0000000001,\n Unit::Lightyear => 9.461e+15,\n};\n\n\n \n \n \n \nMultiple Assignments\nIn Ruby you can assign multiple values to variables in one step:\n\n\n\ndef values\n [1, 2, 3]\nend\n\none, two, three = values\n\nIn Rust, you can only decompose tuples into tuples, but not a vector into a tuple for example.\nSo this will work:\n\n\n\nlet (one, two, three) = (1, 2, 3);\n\nBut this won't:\n\n\n\nlet (one, two, three) = [1, 2, 3];\n// ^^^^^^^^^^^^^^^^^ expected array of 3 elements, found tuple\n\nNeither will this:\n\n\n\nlet (one, two, three) = [1, 2, 3].iter().collect();\n// a collection of type `(_, _, _)` cannot be built from an iterator over elements of type `&{integer}`\n\nBut with nightly Rust, you can now do this:\n\n\n\nlet [one, two, three] = [1, 2, 3];\n\nOn the other hand, there's a lot more you can do with destructuring apart from multiple assignments. You can write beautiful, ergonomic code using pattern syntax.\n\n\n\nlet x = 4;\nlet y = false;\n\nmatch x {\n 4 | 5 | 6 if y => println!("yes"),\n _ => println!("no"),\n}\n\nTo quote The Book:\n\nThis prints no since the if condition applies to the whole pattern 4 | 5 | 6, not only to the last value 6.\n\n\n \n \n \n \nString interpolation\nRuby has extensive string interpolation support.\n\n\n\nprogramming_language = "Ruby"\n"#{programming_language} is a beautiful programming language"\n\nThis can be translated like so:\n\n\n\nlet programming_language = "Rust";\nformat!("{} is also a beautiful programming language", programming_language);\n\nNamed arguments are also possible, albeit much less common:\n\n\n\nprintln!("{language} is also a beautiful programming language", language="Rust");\n\nRust's println!() syntax is even more extensive than Ruby's. Check the docs if you're curious about what else you can do.\n\n \n \n \n \nThat’s it!\nRuby comes with syntactic sugar for many common usage patterns, which allows for very elegant code.\nLow-level programming and raw performance are no primary goals of the language.\nIf you do need that, Rust might be a good fit, because it provides fine-grained hardware control with comparable ergonomics.\nIf in doubt, Rust favors explicitness, though; it eschews magic.\nDid I whet your appetite for idiomatic Rust? Have a look at this Github project. I'd be thankful for contributions.\n\n \n \n \n \nFootnotes\n1. Thanks to Florian Gilcher for the hint.↩\n2. Thanks to masklin for pointing out multiple inaccuracies.↩\n3. In the first version, I sait that ok() would convert a Result into a boolean, which was wrong. Thanks to isaacg for the correction.↩\n" }, { "title": "Making Myself Obsolete", "url": "https://endler.dev/2017/obsolete/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n The Stegosaurus had better days 150 million years ago.\n \n \n \n \n \n Source: Paleontologists once thought it had a brain in its butt.\n \n \n \n\nIn December 2015 I was looking for static analysis tools to integrate into trivago's CI process.\nThe idea was to detect typical programming mistakes automatically.\nThat's quite a common thing, and there are lots of helpful tools out there which fit the bill.\nSo I looked for a list of tools...\nTo my surprise, the only list I found was on Wikipedia β€” and it was outdated.\nThere was no such project on Github, where most modern static analysis tools were hosted.\nWithout overthinking it, I opened up my editor and wrote down a few tools I found through my initial research. After that, I pushed the list to Github.\nI called the project Awesome Static Analysis.\nFast forward two years and the list has grown quite a bit.\nSo far, it has 75 contributors, 277 forks and received over 2000 stars. (Thanks for all the support!)\n(Update May 2018: 91 contributors, 363 forks, over 3000 stars)\nAround 1000 unique visitors find the list every week. Not much by any means, but I feel obliged to keep it up-to-date\nbecause it has become an essential source of information for many people.\nIt now lists around 300 tools for static analysis. Everything from Ada to TypeScript is on there.\nWhat I find particularly motivating is, that now the authors themselves create pull requests to add their tools!\nThere was one problem though: The list of pull requests got longer and longer, as I was busy doing other things.\n\n\n \n \n \n \nAdding contributors\nI always try to make team members out of regular contributors. My friend and colleague Andy Grunwald as well as Ouroboros Chrysopoeia are both valuable collaborators. They help me weed out new PRs whenever they find the time.\nBut let's face it: checking the pull requests is a dull, manual task.\nWhat needs to be checked for each new tool can be summarized like this:\n\nFormatting rules are satisfied\nProject URL is reachable\nLicense annotation is correct\nTools of each section are alphabetically ordered\nDescription is not too long\n\nI guess it's obvious what we should do with that checklist: automate it!\n\n \n \n \n \nA linter for linting linters\nSo why not write an analysis tool, which checks our list of analysis tools!\nWhat sounds pretty meta, is actually pretty straightforward.\nWith every pull request, we trigger our bot, which checks the above rules and responds with a result.\nThe first step was to read the Github documentation about building a CI server.\nJust for fun, I wanted to create the bot in Rust.\nThe two most popular Github clients for Rust were github-rs and hubcaps.\nBoth looked pretty neat, but then I found afterparty, a "Github webhook server".\nThe example looked fabulous:\n\n#[macro_use]\nextern crate log;\nextern crate env_logger;\nextern crate afterparty;\nextern crate hyper;\n\nuse afterparty::{Delivery, Hub};\n\nuse hyper::Server;\n\npub fn main() {\n env_logger::init().unwrap();\n let addr = format!("{}", 4567);\n let mut hub = Hub::new();\n hub.handle("pull_request", |delivery: &Delivery| {\n match delivery.payload {\n Event::PullRequest { ref action, ref sender, .. } => {\n // TODO: My code here!\n println!("sender {} action {}", sender.login, action)\n }\n _ => (),\n }\n });\n let srvc = Server::http(&addr[..])\n .unwrap()\n .handle(hub);\n println!("listening on {}", addr);\n srvc.unwrap();\n}\n\nThis allowed me to focus on the actual analysis code,\nwhich makes for a pretty boring read. It mechanically checks for the things mentioned above and could be written in any language.\nIf you want to have a look (or even contribute!), check out the repo.\n\n \n \n \n \nTalking to Github\nAfter the analysis code was done, I had a bot, running locally, waiting for incoming pull requests.\nBut how could I talk to Github?\nI found out, that I should use the Status API\nand send a POST request to /repos/mre/awesome-static-analysis/statuses/:sha\n(:sha is the commit ID that points to the HEAD of the pull request):\n\n{\n "state": "success",\n "description": "The build succeeded!"\n}\n\nI could have used one of the existing Rust Github clients, but I decided to write a simple function to update the pull request status code.\n\nfn set_status(status: Status, desc: String, repo: &str, sha: &str) -> Result<reqwest::Response> {\n let token = env::var("GITHUB_TOKEN")?;\n let client = reqwest::Client::new();\n let mut params = HashMap::new();\n params.insert("state", format!("{}", status));\n params.insert("description", desc);\n println!("Sending status: {:#?}", params);\n\n let status_url = format!("https://api.github.com/repos/{}/statuses/{}", repo, sha);\n println!("Status url: {}", status_url);\n Ok(client\n .request(\n reqwest::Method::Post,\n &format!(\n "{}?access_token={}",\n status_url,\n token,\n ),\n )\n .json(&params)\n .send()?)\n}\n\nYou can see that I pass in a Github token from the environment and then I send the JSON payload as a post request using the reqwest library.\nThat turned out to become a problem in the end: while afterparty was using version 0.9 of hyper, reqwest was using 0.11. Unfortunately, these two versions depend on a different build of the openssl-sys bindings. That's a well-known problem and the only way to fix it, is to resolve the conflict.\nI was stuck for a while, but then I saw, that there was an open pull request to upgrade afterparty to hyper 0.10.\nSo inside my Cargo.toml, I locked the version of afterparty to the version of the pull request:\n\n[dependencies]\nafterparty = { git = "https://github.com/ms705/afterparty" }\n\nThis fixed the build, and I could finally move on.\n\n \n \n \n \nDeployment\nI needed a place to host the bot.\nPreferably for free, as it was a non-profit Open Source project.\nAlso, the provider would have to run binaries.\nFor quite some time, I was following a product named zeit.\nIt runs any Docker container using an intuitive command line interface called now.\nI fell in love the first time I saw their demo on the site, so I wanted to give it a try.\nSo I added a multi-stage Dockerfile to my project:\n\nFROM rust as builder\nCOPY . /usr/src/app\nWORKDIR /usr/src/app\nRUN cargo build --release\n\nFROM debian:stretch\nRUN apt update \\\n && apt install -y libssl1.1 ca-certificates \\\n && apt clean -y \\\n && apt autoclean -y \\\n && apt autoremove -y\nCOPY --from=builder target/release/check .\nEXPOSE 4567\nENTRYPOINT ["./check"]\nCMD ["--help"]\n\nThe first part would build a static binary, the second part would run it at container startup.\nWell, that didn't work, because zeit does not support multi-stage builds yet.\nThe workaround was to split up the Dockerfile into two and connect them both with a Makefile. Makefiles are pretty powerful, you know?\nWith that, I had all the parts for deployment together.\n\n# Build Rust binary for Linux\ndocker run --rm -v $(CURDIR):/usr/src/ci -w /usr/src/ci rust cargo build --release\n\n# Deploy Docker images built from the local Dockerfile\nnow deploy --force --public -e GITHUB_TOKEN=${GITHUB_TOKEN}\n\n# Set domain name of new build to `check.now.sh`\n# (The deployment URL was copied to the clipboard and is retrieved with pbpaste on macOS)\nnow alias `pbpaste` check.now.sh\n\nHere's the output of the deploy using now:\n\n> Deploying ~/Code/private/awesome-static-analysis-ci/deploy\n> Ready! https://deploy-sjbiykfvtx.now.sh (copied to clipboard) [2s]\n> Initializing…\n> Initializing…\n> Building\n> β–² docker build\nSending build context to Docker daemon 2.048 kBkB\n> Step 1 : FROM mre0/ci:latest\n> latest: Pulling from mre0/ci\n> ...\n> Digest: sha256:5ad07c12184755b84ca1b587e91b97c30f7d547e76628645a2c23dc1d9d3fd4b\n> Status: Downloaded newer image for mre0/ci:latest\n> ---> 8ee1b20de28b\n> Successfully built 8ee1b20de28b\n> β–² Storing image\n> β–² Deploying image\n> β–² Container started\n> listening on\n> Deployment complete!\n\nThe last step was to add check.now.sh as a webhook inside the awesome-static-analysis project settings.\nNow, whenever a new pull request is coming in, you see that little bot getting active!\n\n\n \n \n \n \nOutcome and future plans\nI am very pleased with my choice of tools: afterparty saved me from a lot of manual work, while zeit made deployment really easy.\nIt feels like Amazon Lambda on steroids.\nIf you look at the code and the commits for my bot, you can see all my little missteps, until I got everything just right. Turns out, parsing human-readable text is tedious.\nTherefore I was thinking about turning the list of analysis tools into a structured format like YAML. This would greatly simplify the parsing and have the added benefit of having a machine-readable list of tools that can be used for other projects.\n\n \n \n \n \nUpdate May 2018\nWhile attending the WeAreDevelopers conference in Vienna (can recommend that), I moved the CI pipeline from zeit.co to Travis CI.\nThe reason was, that I wanted the linting code next to the project, which greatly simplified things.\nFirst and foremost I don't need the web request handling code anymore, because travis takes care of that.\nIf you like, you can compare the old and the new version.\n" }, { "title": "Modern Day Annoyances - Digital Clocks", "url": "https://endler.dev/2017/digitial-clocks/", "body": "This morning I woke up to the beeping noise of our oven's alarm clock.\nThe reason was that I tried to correct the oven's local time the day before β€” and I pushed the wrong buttons.\nAs a result I didn't set the correct time, instead, I set a cooking timer... and that's what woke me up today.\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n\n\n \n \n \n \nLet's add a clock to the microwave!\nOn occasions like these, I wonder why there's a digital clock on every single household device these days.\nThey're integrated into microwaves, fridges, ovens, dishwashers, dryers, mixers β€” and that's just the kitchen!\nThere is an inflation of digital clocks on modern-day devices.\nA lot of times I was wondering why that is the case. Here's my best guess:\nIt's easier to add a useless digital clock to the design than to leave it out.\nSay you are the engineer responsible for the control panel of a run-of-the-mill microwave.\nThe microwave chip comes with a digital timer, which is perfect for showing the remaining time until the food is warmed up.\nNow the question is, what will the timer show when you don't want to heat anything?\nWell, why not show the current time?\nIt's unobtrusive and adds value.\nExcept that these digital clocks can be quite annoying:\n\nThey run out of sync and show the wrong time.\nThey get reset when being plugged off or there's a power outage. (That's the dreaded, blinking 00:00 we all learned to love.)\nThey don't automatically switch between summer and winter time (hey Germany!).\n\nThat's why I constantly need to look after those clocks.\nLet me tell you a secret:\nWhen I'm not warming stuff in the oven, I don't want it to tell me the local time. I want the stove to be off.\n\n \n \n \n \nWhy I have trouble setting the clock on our oven\nOur oven has three buttons related to time: plus, minus and a clock symbol.\nTo set the time, you push the clock symbol. An arrow appears and the display changes to 00:00. You press time again and another arrow appears.\nPressing it two more times shows a blinking clock symbol. Then you can use the + and - buttons to adjust the time. After that, you wait to confirm.\nEasy!\nThe problem is, there is no immediate relationship between the controls and the result in the world.\nThe underlying concept is called mapping and is prevalent in interface design.\nTo add some functionality to a device you have two options:\n\nAdd more buttons.\nTeach an existing button a new trick.\n\nOption 1 might dilute your beautiful design, while option 2 might mean frustration for your users.\nNeither option is appealing.\nOur oven maps multiple functions to the same button.\nBut the most annoying thing is, that each device has a different mapping.\nLearning to set the time on my oven won't help me with the dishwasher, which sports an entirely different interface!\n\n \n \n \n \nTakeaways\nGood industrial designs are few and far between.\nA clock on your product will most likely not add any additional value.\nIn the best case it might be an annoyance, in the worst case it's harmfully misleading.\nWhen given a choice, I prefer home appliances without clocks.\nLooking at today's market, that's harder than it sounds.\nArguably, a device with a clock is cheaper than one without; just because the ones with timers get produced more often.\nNow I can understand why it took Steve Jobs two weeks to decide on a washing machine:\n\nWe spent some time in our family talking about what's the trade-off we want to make.\nWe spent about two weeks talking about this. Every night at the dinner table.\n\nHe chose a Miele Washing machine in the end - without a digital clock, I assume.\n" }, { "title": "Learn Some Rust During Hacktoberfest", "url": "https://endler.dev/2017/hacktoberfest/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n Dirndl, Lederhose, Brezn, Beer, Rust\n \n \n \n \n \n Source: Designed by Freepik\n \n \n \n\nOctober is the perfect time to contribute to Open Source β€” at least according to Github and DigitalOcean.\nBecause that's when they organize Hacktoberfest, a global event where you get a free shirt and lots of street cred for creating pull requests. Read the official announcement here.\nSome people think they cannot contribute anything of value. Either because they lack the programming skills or because they don't know where to start.\nThis guide is trying to change that!\nLet me show you, how everybody can contribute code to Rust, a safe systems programming language.\nI was inspired to write this by a tweet from llogiq.\n\n \n \n \n \n1. Find a great Rust project to work on\nWe all want our work to be appreciated.\nTherefore I suggest to start contributing to medium-sized projects, because they gained some momentum but are still driven by a small number of maintainers, so help is always welcome. By contrast, tiny projects are mostly useful to the original author only, while large projects can be intimidating at first and have stricter guidelines.\nFor now, let's look at repositories with 5-100 stars, which were updated within this year.\nGithub supports advanced search options based on Lucene syntax.\n\nlanguage:Rust stars:5..100 pushed:>2017-01-01\n\nHere's a list of projects, which match this filter.\n\n \n \n \n \n2. Install the Rust toolchain\nTo start contributing, we need a working Rust compiler and the cargo package manager.\nFortunately, the installation should be straightforward.\nI recommend rustup for that.\nRun the following command in your terminal, then follow the onscreen instructions.\n\ncurl https://sh.rustup.rs -sSf | sh\n\nIf you're unsure, just accept the defaults.\nAfter the installation is done, we also need to get the nightly version of the compiler for later.\n\nrustup install nightly\n\nQuestions so far? Find more detailed installation instructions here.\n\n \n \n \n \n3. Fork the project and clone it to your computer\nFirst, click on the little fork button on the top right of the Github project page. Then clone your fork to your computer.\n\ngit clone git@github.com:yourusername/project.git\n\nFor more detailed instructions, go here.\n\n \n \n \n \n4. Does it build?\nBefore we start modifying the codebase, we should make sure that it is in a workable state.\nThe following commands should work right away from inside the project folder.\n\ncargo build\ncargo test\n\nIf not, you might want to consult the README for further instructions. (But feel free to choose another project.)\n\n \n \n \n \n5. The magic sauce\nHere's the trick: we use a linter called clippy to show us improvement areas in any Rust codebase.\nTo get clippy, install it like so:\n\ncargo +nightly install clippy\n\nAfterwards, run it from the project root as often as you like.\n\nrustup run nightly cargo clippy\n\nThis should give you actionable information on how to improve the codebase.\nHere's some sample output:\n\nwarning: useless use of `format!`\n --> src/mach/header.rs:420:49\n |\n420 | let error = error::Error::Malformed(format!("bytes size is smaller than an Mach-o header"));\n | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n |\n = note: #[warn(useless_format)] on by default\n = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#useless_format\n\nwarning: this expression borrows a reference that is immediately dereferenced by the compiler\n --> src/mach/header.rs:423:36\n |\n423 | let magic = mach::peek(&bytes, 0)?;\n | ^^^^^^ help: change this to: `bytes`\n |\n = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#needless_borrow\n\nJust try some of the suggestions and see if the project still compiles and the tests still pass.\nCheck out the links to the documentation in the help section to learn more.\nStart small to make your changes easier to review.\n\n \n \n \n \n6. Creating a Pull Request\nIf you're happy with your changes, now is the time to publish them!\nIt's best to create a new branch for your changes and then push it to your fork.\n\ngit checkout -b codestyle\ngit commit -am "Minor codestyle fixes"\ngit push --set-upstream origin codestyle\n\nAfterwards, go to the homepage of your fork on Github.\nThere should be a button titled Compare & pull request.\nPlease add a meaningful description and then submit the pull request.\nCongratulations! You've contributed to the Rust ecosystem. Thank you! πŸŽ‰\n\n \n \n \n \nTrophy case\n\nm4b/goblin\nfitzgen/cpp_demangle\nfdehau/tui-rs\nchristophertrml/rs-natural\n\n\n \n \n \n \nBonus!\nIf all of the manual fixing and checking sounds too dull, you can automate step number 5 using rustfix by Pascal Hertleif (@killercup):\n\nrustfix --yolo && cargo check\n" }, { "title": "A Little Story About the `yes` Unix Command", "url": "https://endler.dev/2017/yes/", "body": "What's the simplest Unix command you know?\nThere's echo, which prints a string to stdout and true, which always terminates with an exit code of 0.\nAmong the rows of simple Unix commands, there's also yes.\nIf you run it without arguments, you get an infinite stream of y's, separated by a newline:\n\ny\ny\ny\ny\n(...you get the idea)\n\nWhat seems to be pointless in the beginning turns out to be pretty helpful :\n\nyes | sh boring_installation.sh\n\nEver installed a program, which required you to type "y" and hit enter to keep going?\nyes to the rescue! It will carefully fulfill its duty, so you can keep watching Pootie Tang.\n\n \n \n \n \nWriting yes\nHere's a basic version in... uhm... BASIC.\n\n10 PRINT "y"\n20 GOTO 10\n\nAnd here's the same thing in Python:\n\nwhile True:\n print("y")\n\nSimple, eh? Not so quick!\nTurns out, that program is quite slow.\n\npython yes.py | pv -r > /dev/null\n[4.17MiB/s]\n\nCompare that with the built-in version on my Mac:\n\nyes | pv -r > /dev/null\n[34.2MiB/s]\n\nSo I tried to write a quicker version in Rust. Here's my first attempt:\n\nuse std::env;\n\nfn main() {\n let expletive = env::args().nth(1).unwrap_or("y".into());\n loop {\n println!("{}", expletive);\n }\n}\n\nSome explanations:\n\nThe string we want to print in a loop is the first command line parameter and is named expletive. I learned this word from the yes manpage.\nI use unwrap_or to get the expletive from the parameters. In case the parameter is not set, we use "y" as a default.\nThe default parameter gets converted from a string slice (&str) into an owned string on the heap (String) using into().\n\nLet's test it.\n\ncargo run --release | pv -r > /dev/null\n Compiling yes v0.1.0\n Finished release [optimized] target(s) in 1.0 secs\n Running `target/release/yes`\n[2.35MiB/s]\n\nWhoops, that doesn't look any better. It's even slower than the Python version!\nThat caught my attention, so I looked around for the source code of a C implementation.\nHere's the very first version of the program, released with Version 7 Unix and famously authored by Ken Thompson on Jan 10, 1979:\n\nmain(argc, argv)\nchar **argv;\n{\n for (;;)\n printf("%s\\n", argc>1? argv[1]: "y");\n}\n\nNo magic here.\nCompare that to the 128-line-version from the GNU coreutils, which is mirrored on Github. After 25 years, it is still under active development!\nThe last code change happened around a year ago.\nThat's quite fast:\n\n# brew install coreutils\ngyes | pv -r > /dev/null\n[854MiB/s]\n\nThe important part is at the end:\n\n/* Repeatedly output the buffer until there is a write error; then fail. */\nwhile (full_write (STDOUT_FILENO, buf, bufused) == bufused)\n continue;\n\nAha! So they simply use a buffer to make write operations faster.\nThe buffer size is defined by a constant named BUFSIZ, which gets chosen on each system so as to make I/O efficient (see here).\nOn my system, that was defined as 1024 bytes. I actually had better performance with 8192 bytes.\nI've extended my Rust program:\n\nuse std::env;\nuse std::io::{self, BufWriter, Write};\n\nconst BUFSIZE: usize = 8192;\n\nfn main() {\n let expletive = env::args().nth(1).unwrap_or("y".into());\n let mut writer = BufWriter::with_capacity(BUFSIZE, io::stdout());\n loop {\n writeln!(writer, "{}", expletive).unwrap();\n }\n}\n\nThe important part is, that the buffer size is a multiple of four, to ensure memory alignment.\nRunning that gave me 51.3MiB/s.\nFaster than the version, which comes with my system, but still way slower than the results from this Reddit post that I found, where the author talks about 10.2GiB/s.\n\n \n \n \n \nUpdate\nOnce again, the Rust community did not disappoint.\nAs soon as this post hit the Rust subreddit, user nwydo pointed out a previous discussion on the same topic.\nHere's their optimized code, that breaks the 3GB/s mark on my machine:\n\nuse std::env;\nuse std::io::{self, Write};\nuse std::process;\nuse std::borrow::Cow;\n\nuse std::ffi::OsString;\npub const BUFFER_CAPACITY: usize = 64 * 1024;\n\npub fn to_bytes(os_str: OsString) -> Vec<u8> {\n use std::os::unix::ffi::OsStringExt;\n os_str.into_vec()\n}\n\nfn fill_up_buffer<'a>(buffer: &'a mut [u8], output: &'a [u8]) -> &'a [u8] {\n if output.len() > buffer.len() / 2 {\n return output;\n }\n\n let mut buffer_size = output.len();\n buffer[..buffer_size].clone_from_slice(output);\n\n while buffer_size < buffer.len() / 2 {\n let (left, right) = buffer.split_at_mut(buffer_size);\n right[..buffer_size].clone_from_slice(left);\n buffer_size *= 2;\n }\n\n &buffer[..buffer_size]\n}\n\nfn write(output: &[u8]) {\n let stdout = io::stdout();\n let mut locked = stdout.lock();\n let mut buffer = [0u8; BUFFER_CAPACITY];\n\n let filled = fill_up_buffer(&mut buffer, output);\n while locked.write_all(filled).is_ok() {}\n}\n\nfn main() {\n write(&env::args_os().nth(1).map(to_bytes).map_or(\n Cow::Borrowed(\n &b"y\\n"[..],\n ),\n |mut arg| {\n arg.push(b'\\n');\n Cow::Owned(arg)\n },\n ));\n process::exit(1);\n}\n\nNow that's a whole different ballgame!\n\nWe prepare a filled string buffer, which will be reused for each loop.\nStdout is protected by a lock. So, instead of constantly acquiring and releasing it, we keep it all the time.\nWe use a the platform-native std::ffi::OsString and std::borrow::Cow to avoid unnecessary allocations.\n\nThe only thing that I could contribute was removing an unnecessary mut. πŸ˜…\n\n \n \n \n \nLessons learned\nThe trivial program yes turns out not to be so trivial after all.\nIt uses output buffering and memory alignment to improve performance.\nRe-implementing Unix tools is fun and makes me appreciate the nifty tricks,\nwhich make our computers fast.\n" }, { "title": "Lightning Fast Image Previews with Pure CSS and LQIP", "url": "https://endler.dev/2017/image-previews/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n Source: Adapted from Freepik \n \n \n \n\nMy website is reasonably fast.\nI hope that every page load feels snappy, no matter your device or location.\nThat should not come as a surprise. After all, I'm just using plain HTML and CSS.\nJavaScript is avoided whenever possible.\nThere was one thing left, which really annoyed me: layout reflow after images got loaded.\nThe problem is, that the image dimensions are not known when the text is ready to be displayed.\nAs a result, the text will be pushed down on the screen as soon as an image is loaded above.\nAlso, while an image is loading, there is no preview, just blank space.\nHere's what that looks like on a slower connection:\n\nI could fix that, by hardcoding the image width and height, but that would be tedious and error-prone.\nAnd there would be no preview.\nSo I was wondering, what others were doing. πŸ€”\n\n \n \n \n \nTiny image thumbnails\nI vaguely remembered, that Facebook uses tiny preview thumbnails in their mobile app.\nThey extract the quantization table from the JPEG header to render the preview. This information\nis stored on the client, so it doesn't need to be downloaded every time.\nUnfortunately, this approach requires full control over the image encoder.\nIt works for apps, but hardly for websites.\nThe search continued.\nUntil my colleague Tobias Baldauf introduced me to LQIP (Low-Quality Image Placeholders).\nHere's the idea:\n\nLoad the page including inlined, low-quality image thumbnails.\nOnce the page is fully loaded (e.g. when the onload event is fired), lazy load full quality images.\n\nUnfortunately, this technique requires JavaScript.\nNevertheless, I liked the idea, so I started experimenting with different image sizes and formats. My goal was to create the smallest thumbnails using any standard image format.\n\n \n \n \n \nBenchmark\nHere are 15 pixel wide thumbnails encoded in different file formats:\n\nI used different tools to create the thumbnails.\nFor JPEG and PNG encoding, I used svgexport.\n\nsvgexport img.svg img.png "svg{background:white;}" 15: 1%\n\nFor webp, I used cwebp:\n\ncwebp img.png -o img.webp\n\nThe gif was converted using an online tool and optimized using gifsicle:\n\ngifsicle -O3 < img.gif > img_mini.gif\n\n \n \n \n \nComparison\nWebP is the smallest, but it's not supported by all browsers.\nGif was second, but when resizing the image and applying the blur filter, I was not happy with the result.\nIn the end, I settled for PNG, which provided an excellent tradeoff between size and quality.\nI optimized the images even further using oxipng, which supports zopfli compression.\nWith that, I end up with thumbnails of around 300-400 bytes in size.\nI integrated the thumbnail creation process into my build toolchain for the blog.\nThe actual code to create the images is rather boring.\nIf you really want to have a look, it's on Github.\n\n \n \n \n \nAvoiding JavaScript\nHere is the skeleton HTML for the image previews:\n\n<figure>\n <div class="loader">\n <object data="image.svg" type="image/svg+xml"></object>\n <img class="frozen" src="data:image/png;base64,..." />\n </div>\n</figure>\n\nThe trick is to wrap both the full-size image and the preview image into a loader div,\nwhich gets a width: auto CSS attribute:\n\n.loader {\n position: relative;\n overflow: hidden;\n width: auto;\n}\n\nI wrap the SVG into an object tag instead of using an img element.\nThis has the benefit, that I can show a placeholder in case the SVG can't be loaded.\nI position the object at the top left of the loader div.\n\n.loader object {\n position: absolute;\n}\n\n.loader img,\n.loader object {\n display: block;\n top: 0;\n left: 0;\n width: 100%;\n}\n\nHere's the placeholder hack including some references:\n\n/* https://stackoverflow.com/a/29111371/270334 */\n/* https://stackoverflow.com/a/32928240/270334 */\nobject {\n position: relative;\n float: left;\n display: block;\n\n &::after {\n position: absolute;\n top: 0;\n left: 0;\n display: block;\n width: 1000px;\n height: 1000px;\n content: "";\n background: #efefef;\n }\n}\n\nThe last part is the handling of the thumbnails.\nLike most other sites, I decided to apply a blur filter.\nIn a way, it looks like the image is frozen, so that's what I called the CSS selector.\nI also applied a scaling transformation to achieve sharp borders.\n\n.frozen {\n -webkit-filter: blur(8px);\n -moz-filter: blur(8px);\n -o-filter: blur(8px);\n -ms-filter: blur(8px);\n filter: blur(8px);\n transform: scale(1.04);\n animation: 0.2s ease-in 0.4s 1 forwards fade;\n width: 100%;\n}\n\n@keyframes fade {\n 0% {\n opacity: 1;\n }\n 100% {\n opacity: 0;\n }\n}\n\nI use CSS animations instead of JavaScript.\nThe duration of the animation is based on the 95% percentile load time of all visitors of the page. Although it's just an approximation, this should work for most readers.\n\n \n \n \n \nResult\n\nNo JavaScript needed\nWorks on all modern browsers\nSupports a fallback in case the main image can't be loaded\nTiny overhead\n\n\n \n \n \n \nResources\n\nIntroducing LQIP – Low Quality Image Placeholders\nHow Medium does progressive image loading\nSQIP, a new preview technique using pure SVG\n\n" }, { "title": "Go vs Rust? Choose Go.", "url": "https://endler.dev/2017/go-vs-rust/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n Source: Gopher designed with Gopherize.me. Gears designed by Freepik\n \n \n \n\n"Rust or Go, which one should I choose?" is a question I get quite often.\nBoth languages seem to be competing for the same user base and they both seem to be\nsystems programming languages, so there must be a clear winner, right?\ntl;dr: It's not so easy. Both languages have a different scope. Golang shines for writing microservices and for typical "DevOps" tasks, but it is not a systems programming language. Rust is stronger for tasks where concurrency, safety and/or performance are important; but it has a steeper learning curve than Go.\n\n \n \n \n \nGo: practical, pragmatic, plain\n\nI don't think Go is an elegant language. Its biggest feature is simplicity.\nGo is not even a systems programming language. While it's great for writing microservices and tooling around backend infrastructure, I would not want to write a kernel or a memory allocator with it.\nBut with Go, you get things done β€” fast.\nGo is one of the most productive languages I've ever worked with.\nThe mantra is: solve real problems today.\n\n \n \n \n \nRust's strong guarantees come at a cost\n\nRust in comparison is hard. It took me many months to become somewhat productive.\nYou need to invest a serious amount of time to see any benefit.\nRust is already a powerful language and it gets stronger every day.\nIt feels much more like a pragmatic Haskell to me than a safer C.\nDon't get me wrong: I love Rust, and it helped me become a better programmer. It is certainly a nice language to learn. The big question is, if it is the right choice for your next major project.\nHere's the thing: if you choose Rust, usually you need the guarantees, that the language provides:\n\nSafety against Null pointers, race conditions and all sorts of low-level threats.\nPredictable runtime behavior (zero cost abstractions and no garbage collector).\n(Almost) total control over the hardware (memory layout, processor features).\nSeamless interoperability with other languages.\n\nIf you don't require any of these features, Rust might be a poor choice for your next project.\nThat's because these guarantees come with a cost: ramp-up time.\nYou'll need to unlearn bad habits and learn new concepts.\nChances are, you will fight with the borrow checker a lot when you start out.\n\n \n \n \n \nCase-study: Primality by trial division\nLet's say, you want to check if a number is prime.\nThe easiest way is to check if we can divide the number by any smaller natural number (without a remainder). If not, we found a prime number! This approach is called trial division.\nHere's how to do that in Golang (courtesy of Rosetta Code):\n\nfunc IsPrime(n int) bool {\n\tif n < 0 {\n\t\tn = -n\n\t}\n\tswitch {\n\tcase n < 2:\n\t\treturn false\n\tdefault:\n\t\tfor i := 2; i < n; i++ {\n\t\t\tif n%i == 0 {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t}\n\treturn true\n}\n\nAnd here's the same thing in Rust:\n\npub fn is_prime(n: u64) -> bool {\n match n {\n 0...1 => false,\n _ => {\n for d in 2..n {\n if n % d == 0 {\n return false;\n }\n }\n true\n }\n }\n}\n\nAt first sight, both solutions look pretty similar.\nBut if we look closer, we can spot some differences.\n\nIn Go, we use a simple switch-case statement. In Rust, we use a match statement, which is much more powerful.\nIn Go, we use a simple for-loop to iterate over the numbers 2 to n. In Rust, we use a range expression (2..n).\nIn Go, we use two return statements, in Rust we have one return expression. In general, most things in Rust are expressions, which can be returned and assigned to a variable. Read more about expressions here.\n\nIn many areas, Rust is more functional than Golang. You could rewrite the above code using the any method, which is implemented for Range.\n\nfn is_prime(n: u64) -> bool {\n match n {\n 0...1 => false,\n _ => !(2..n).any(|d| n % d == 0),\n }\n}\n\nIt might seem a little alien at first, but it will become second-nature after a while.\nThis was just a quick example, of course. I suggest, you browse some code on Rosetta Code to get a better feeling for both languages.\n\n \n \n \n \nCase study: Finding duplicate words in text files\nIf you're more like a visual type, here is a video where I write a simple\nconcurrent program in Go and Rust to compare both languages:\n\n\n \n \n \n \n \n \n \n\n\n\n document.addEventListener(\"DOMContentLoaded\", function() {\n lightEmbedInit();\n });\n\n\n \n \n \n \nSome things I prefer in Go\n\nFast compile times\nPragmatic problem-solving approach\nNice ecosystem for typical DevOps tasks\nBatteries-included standard-library\nIDE support\nSimple error handling\nThe mascot πŸ˜‰\n\n\n \n \n \n \nSome things I prefer in Rust\n\nSafety: No null pointers, no data races,...\nFine-grained system control\nIncredible runtime speed (comparable with C/C++)\nZero-cost abstractions\nAwesome, open-minded community\nSimple package management with cargo\nSupport for Generics in form of traits\nC interop and FFI\n\n\n \n \n \n \nConclusion\n99% of the time, Go is "good enough" and that 1% where it isn't, you'll know.\nAnd then take a look at Rust, because the two languages complement each other pretty well.\nIf you're interested in hands-on Rust consulting, pick a date from my\ncalendar and we can talk about how I can help.\nAfter all is said and done, Rust and Go are not really competitors.\n" }, { "title": "Afraid of Makefiles? Don't be!", "url": "https://endler.dev/2017/makefiles/", "body": "\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n What do clothes have to do with Makefiles? Find out in this post.\n \n \n \n \n \n Source: Illustration by Anindyanfitri - Freepik.com\n \n \n \n\nIn the last few years, I've had the pleasure to work with a lot of talented Software Engineers.\nOne thing that struck me is, that many of them did not have any working knowledge of Makefiles\nand why they are useful.\nWhen faced with the task to automate a build process, they often roll their own shell scripts.\nCommon culprits are called build.sh or run.sh or doall.sh etc.\nThey implement the same basic functionality over and over again:\n\nParsing input parameters and environment variables.\nManually managing dependencies between build steps.\nError handling... maybe.\n\nAlong the way, they keep making the same basic mistakes:\n\nIncorrectly handling input parameters and environment variables.\nMissing dependencies between build steps.\nForgetting to handle errors and β€” even worse β€” carrying on with the program execution.\n\n\n \n \n \n \nMakefiles are scary!\nIf you think that make is scary, you probably think of complicated build machinery for big software projects.\nIt doesn't need to be that way. Let's hear what the author of make, Stuart Feldman has to say:\n\nIt began with an elaborate idea of a dependency analyzer, boiled down to something much simpler, and turned into Make that weekend. Use of tools that were still wet was part of the culture. Makefiles were text files, not magically encoded binaries because that was the Unix ethos: printable, debuggable, understandable stuff.\nβ€” The Art of Unix Programming (2003)\n\n\n \n \n \n \nMakefiles are simple!\nBefore I leave the house, I need to get dressed.\nI use the same simple routine every time:\nUnderpants, trousers, shirt, pullover, socks, shoes, jacket.\nMost likely you also have a routine, even though yours might be different.\nSome of these steps depend on each other.\nMake is useful for handling dependencies.\nLet's try to express my routine as a Makefile.\n\ndress: trousers shoes jacket\n\t@echo "All done. Let's go outside!"\n\njacket: pullover\n\t@echo "Putting on jacket."\n\npullover: shirt\n\t@echo "Putting on pullover."\n\nshirt:\n\t@echo "Putting on shirt."\n\ntrousers: underpants\n\t@echo "Putting on trousers."\n\nunderpants:\n\t@echo "Putting on underpants."\n\nshoes: socks\n\t@echo "Putting on shoes."\n\nsocks: pullover\n\t@echo "Putting on socks."\n\nIf we execute the Makefile, we get the following output:\n\n$ make dress\nPutting on underpants.\nPutting on trousers.\nPutting on shirt.\nPutting on pullover.\nPutting on socks.\nPutting on shoes.\nPutting on jacket.\nAll done. Let's go outside!\n\n \n \n \n \nWhat just happened?\nNoticed how the steps are in the correct order?\nBy plainly writing down the dependencies between the steps, make helps us to execute them correctly.\nEach build step has the following structure:\n\ntarget: [dependencies]\n\t<shell command to execute>\n\t<shell command to execute>\n\t...\n\n\n\nThe first target in a Makefile will be executed by default when we call make.\n\n\nThe order of the targets does not matter.\n\n\nShell commands must be indented with a tab.\n\n\nAdd an @ sign to suppress output of the command that is executed.\n\n\nIf target isn't a file you want to build, please add .PHONY <target> at the end of the build step.\nCommon phony targets are: clean, install, run,...\n\ninstall:\n\tnpm install\n.PHONY: install\n\nOtherwise, if somebody creates an install directory, make will silently fail, because the build target already exists.\n\n\nCongratulations! You've learned 90% of what you need to know about make.\n\n \n \n \n \nNext steps\nReal Makefiles can do much more! They will only build the files that have changed instead of doing a full rebuild.\nAnd they will do as much as possible in parallel.\n" }, { "title": "Of Boxes and Trees - Smart Pointers in Rust", "url": "https://endler.dev/2017/boxes-and-trees/", "body": "Recently, I tried to implement a binary tree data structure in Rust.\nEach binary tree has a root value, a left, and a right subtree.\nI started from this Python implementation, which is quite straightforward.\n\nclass Tree:\n def __init__(self, val, left=None, right=None):\n self.val = val\n self.left = left\n self.right = right\n\nThis allows us to declare a fancy tree object like this:\n\nt = Tree(15,\n Tree(12,\n None,\n Tree(13)),\n Tree(22,\n Tree(18),\n Tree(100)))\n\nAnd the result can be visualized beautifully.\n(Yes I've drawn that myself.)\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A binary search tree representing our data structure\n \n \n \n \n \n\nPorting that code to Rust turned out to be a little... challenging.\nMy first attempt looked quite innocuous.\n\nstruct Tree {\n root: i64,\n left: Tree,\n right: Tree,\n}\n\nThat's pretty much a one-to-one translation of the Python definition β€” but rustc says no.\n\nerror[E0072]: recursive type `Tree` has infinite size\n --> src/main.rs:1:1\n |\n1 | struct Tree {\n | ^^^^^^^^^^^ recursive type has infinite size\n |\n = help: insert indirection (e.g., a `Box`, `Rc`, or `&`) at some point to make `Tree` representable\n\nComing from memory-managed languages (like Python, PHP, or Ruby), I was confused by this.\nThe problem is easy to understand, though.\nComputers have a limited amount of memory.\nIt's the compiler's job to find out how much memory to allocate for each item.\nIn our case, it infers the following:\nA tree is a structure containing an i64, and two trees. Each of these trees is a structure containing an i64, and two trees. Each of these...\nYou get the idea.\n\nTree { i64, Tree, Tree }\nTree { i64, Tree { ... }, Tree { ... } }\n// The next expansion won't fit on the page anymore\n\nSince we don't know how many subtrees our tree will have, there is no way to tell how much memory we need to allocate up front. We'll only know at runtime!\nRust tells us how to fix that: by inserting an indirection like Box, Rc, or &.\nThese are different "pointer types" in Rust. They all point to places in memory. So, instead of knowing the total size of our tree structure, we just know the point in memory where the tree is located. But that's enough to define the tree structure.\nThese pointer types allow us to do that safely and without manual memory management.\nThey all offer different guarantees and you should choose the one that fits your requirements best.\n\n\n& is called a borrow in Rust speech. It's the most common of the three. It's a reference to some place in memory, but it does not own the data it points to. As such, the lifetime of the borrow depends on its owner.\nTherefore we would need to add lifetime parameters here. This can make it tedious to use.\n\nstruct Tree<'a> {\n root: i64,\n left: &'a Tree<'a>,\n right: &'a Tree<'a>,\n}\n\n\nBox is a smart pointer with zero runtime overhead. It owns the data it points to and stores it on the heap.\nWe call it smart because when it goes out of scope, it will first drop the data it points to and then itself. No manual memory management required, which is neat. ✨\n\nstruct Tree {\n root: i64,\n left: Box<Tree>,\n right: Box<Tree>,\n}\n\n\nRc is another smart pointer. It's short for "reference-counting". It keeps track of the number of references to the data structure internally. As soon as the number of references is down to zero, it cleans up after itself.\nChoose Rc if you need to have multiple owners of the same data in one thread.\nFor multithreading, there's also Arc (atomic reference count).\n\nstruct Tree {\n root: i64,\n left: Rc<Tree>,\n right: Rc<Tree>,\n}\n\n\n\n \n \n \n \nPutting the tree into a box\nAll three options are totally valid. Which one you should choose, depends on your use-case.\nA rule of thumb is to keep it simple.\nIn my case, I chose to use a Box, because I did not need any special guarantees.\n\n \n \n \n \nMaking subtrees optional\nThe next problem I faced was that I could not instantiate a tree structure.\nThe left and right subtree have the type Box<Tree>, but at some\npoint I would need an empty subtree.\nIn the Python example, I used None to signal the end of my data structure.\nThanks to Rust's Option type we can do the same:\n\nstruct Tree {\n root: i64,\n left: Option<Box<Tree>>,\n right: Option<Box<Tree>>,\n}\n\nAfter all of this, we can create our first tree:\n\nTree {\n root: 15,\n left: Some(Box::new(Tree {\n root: 12,\n left: None,\n right: Some(Box::new(Tree {\n root: 13,\n left: None,\n right: None,\n })),\n })),\n right: Some(Box::new(Tree {\n root: 22,\n left: Some(Box::new(Tree {\n root: 18,\n left: None,\n right: None,\n })),\n right: Some(Box::new(Tree {\n root: 100,\n left: None,\n right: None,\n })),\n })),\n};\n\nDepending on your point of view, you might say this is either verbose or explicit.\nCompared to the Python version, it looked a bit too cluttered for my taste.\nCan we do better?\nChris McDonald helped me come up with the following representation:\n\nTree::new(15)\n .left(\n Tree::new(12)\n .right(Tree::new(13))\n )\n .right(\n Tree::new(22)\n .left(Tree::new(18))\n .right(Tree::new(100))\n );\n\nTo me, this is much easier on the eye.\nHere's the full tree implementation that makes this possible:\n\n#[derive(Default)]\nstruct Tree {\n root: i64,\n left: Option<Box<Tree>>,\n right: Option<Box<Tree>>,\n}\n\nimpl Tree {\n fn new(root: i64) -> Tree {\n Tree {\n root: root,\n ..Default::default()\n }\n }\n\n fn left(mut self, leaf: Tree) -> Self {\n self.left = Some(Box::new(leaf));\n self\n }\n\n fn right(mut self, leaf: Tree) -> Self {\n self.right = Some(Box::new(leaf));\n self\n }\n}\n\nUpdate: Danny Grein mentioned on Twitter, that\nwe can support the following syntax by implementing From<i64> for Tree:\n\nroot(15)\n .left(\n root(12)\n .right(13)\n )\n .right(\n root(22)\n .left(18)\n .right(100)\n );\n\n \n \n \n \nWhy did it just work in Python?\nNow you might be wondering why our tree implementation worked so flawlessly in Python.\nThe reason is that Python dynamically allocates memory for the tree object at runtime.\nAlso, it wraps everything inside a PyObject, which is kind of similar to Rc from above\nβ€” a reference-counted smart pointer.\nRust is more explicit here. It gives us more flexibility to express our needs\nbut we also need to know about all the possible alternatives to make good use of them.\nMy advice is to stay away from smart pointers if a simple borrow will do.\nIf lifetimes get in the way however or you need additional guarantees like thread-safety, smart pointers are a great addition to your toolkit. The Rust documentation is a good starting point to learn more about smart pointers.\nAlso, read "Idiomatic tree and graph-like structures in Rust" for some clever use of allocators in case your tree needs to be mutable at runtime.\n" }, { "title": "Why Type Systems Matter", "url": "https://endler.dev/2017/why-type-systems-matter/", "body": "I've written most of my code in dynamically typed languages such as Python or PHP. But ever since dabbling with Rust, I've developed a passion for static type systems.\nIt began to feel very natural to me; like a totally new way to express myself.\n\n \n \n \n \nTypes are here to help\nWith types, you communicate your guarantees and expectations. Both, to the machine and other developers. Types express intent.\nAs a programmer, you've probably gained some intuition about types.\n\nsentence = "hello world"\n\nYou might guess that sentence is a string. It's in quotes, after all.\nIt gets a little more tricky if the type gets inferred from some other location.\n\nsentence = x\n\nIs sentence still a string? Uhm... we don't know. It depends on the type of x. Maybe x is a number, and so sentence is also a number? Maybe xused to be a string but during refactoring it is now a byte array? Fun times had by all. πŸŽ‰\nWhat about this one?\n\nfilesize = "5000" # Size in bytes\n\nHere, we express a file size as a string.\nWhile this might work, it's an unsettling idea.\nEven simple calculations might lead to unexpected results:\n\nfile1 = "5000"\nfile2 = "3000"\ntotal = file1 + file2\nprint(total) # prints '50003000'\n\nHow can we fix that?\nWe can safely assume that a file size is always a number.\nTo be more precise, it must be a positive, natural number.\nThere can be no negative file size, and our smallest block of memory is one byte\n(on all but the most obscure systems).\nAnd since we're dealing with a discrete machine here, we know it can only be\na filesize the computer can handle.\nIf we only could express all of this in a precise way...?\nThis is where type systems enter the stage.\nIn Rust, you could define a File type with a field named size.\n\nstruct File {\n name: String,\n size: usize,\n}\n\nThe usize gives you the guarantee to be always big enough to hold any pointer into memory (on 64 bit computers usize = u64).\nNow there is no more ambiguity about the type of size.\nYou can't even create an invalid file object:\n\n// Error: `size` can't be a string.\nlet weird_file = File { name: 123, size: "hello" };\n\nThe type system will prevent invalid state. It will simply not allow you to\nbreak your own rules. It will hold you accountable for your design choices.\nDare I say it: it becomes an extension of your brain.\nAfter some time you start to rely on the type checker. "If it compiles, it runs"\nis a powerful mantra.\n\n \n \n \n \nTypes improve readability and provide context\nConsider the following Python snippet:\n\ndef filter_files(files):\n matches = []\n for file in files:\n if file.status == 0:\n matches.append(file)\n return matches\n\nWhat does 0 represent?\nWe can't say. We lack the context!\nThe story gets a little clearer once we define an enum type like this:\n\nfrom enum import Enum\n\nclass FileStatus(Enum):\n OPEN = 0\n CLOSED = 1\n\nOur example from above becomes\n\ndef filter_files(files):\n matches = []\n for file in files:\n if file.status == FileStatus.OPEN:\n matches.append(file)\n return matches\n\nIn a larger codebase, FileStatus.OPEN is much easier to search for than 0.\nNote: The native enum type was introduced very late in the history of Python. It serves as a nice\nexample of how enhancing the type system can help improve readability.\n\n \n \n \n \nWhen you combine different types, magic happens.\nAll pieces suddenly fall into place when you choose your types wisely. Out of nowhere, the compiler will start\nchecking your design decisions and if all your types work well together. It will point out flaws in your mental model.\nThis gives you a great amount of confidence during refactoring.\nFor example, let's think about sorting things.\nWhen I think of sorting, I first think about a list of numbers:\n\nsorted([1,5,4,3,2]) # [1,2,3,4,5]\n\nThat's the happy path. How about this one?\n\nsorted(1)\n\nOuch. This can't work because 1 is a single number and not a collection!\nIf we forget to check the type before we pass it to sorted, we get an error\nwhile the program runs.\n\nsorted([1, "fish"])\n\nIn Python 2, this would result in [1, 'fish'] (because strings will be compared by length)\nEdit: Reddit user jcdyer3 pointed out that the reason is that when incomparable types are compared, they sort by their type, so all ints will come before all strings. It's a CPython implementation detail).\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n 1 < fish according to Python 2\n \n \n \n \n \n Source: Illustration provided by Freepik\n \n \n \n\nSince Python 3, this throws an Exception.\n\nTypeError: '<' not supported between instances of 'str' and 'int'\n\nMuch better! One less source of error. The problematic thing is though, that this happens at runtime.\nThat's because of Python's dynamic typing.\nWe could have avoided that with a statically typed language.\n\nfn sorted<T>(collection: &mut [T]) where T: PartialOrd {\n // TODO: Sort the collection here.\n}\n\nLooks scary but it really isn't.\nWe define a function named sorted which takes one input parameter named\ncollection.\nThe type of collection consists of four parts:\n\nThe & means that we "borrow" the collection, we don't own it. After the function returns, it will still exist. It won't be cleaned up.\nThe mut means that the collection is mutable. We are allowed to modify it.\n[T] indicates that we expect a list/slice/vector as input. Everything else\nwill be rejected at compile time (before the program even runs).\nPartialOrd is\nthe magic sauce. It is a trait, which is something like an interface. It means that all elements T in the collection must be partially ordered.\n\nAll of this information helps the compiler to prevent us from shooting ourselves in the foot.\nAnd we can understand the inputs and outputs of the function without looking elsewhere.\n\n \n \n \n \nTakeaways\n\nTypes force developers to do their homework and think about the guarantees and limitations of their code.\nDon't think of types as constraints, think of them as a safety net which will protect you from your own flawed mental models.\nAlways choose the type which most precisely expresses your intentions.\nIf there is no perfect type in the standard library, create your own from simpler types.\n\nFollowing these rules, I found that I was magically guided towards the most elegant representation of my ideas.\nMy code became much more idiomatic.\n" }, { "title": "Being a Professional Programmer", "url": "https://endler.dev/2017/professional-programming/", "body": "When I was around 12, I set myself the goal to become a professional programmer.\nI can tell, because at this time I made the conscious decision to use my right hand to control the mouse β€” even though I'm left-handed.\nMy reasoning was, that if I ever had to help out a colleague with a computer problem I sure did not want to move her mouse to the other side before getting started. That would be awkward.\n(Of course I did not foresee the advent of the wireless mouse... As a matter of fact, I still use the right hand out of habit.)\nOne thing I always wanted to know is how a typical workday of a programmer looked like.\nWas I wasting my time by pursuing this career?\nOnly later I found the answer β€” but I had to become a professional programmer myself.\nThis article aims to save you from a few years of uncertainty.\nBefore you dig into this, be sure to read the first part of this series titled "Why I love Programming".\n\n \n \n \n \nWhat's the difference between "professional" and "hobby" programming?\nIn one word: accountability.\nYou are expected to be responsible.\nProgramming in your free time is like throwing a party without having to clean up: pure fun!\nIf you get bored you're free to move on.\nNot so in professional programming, where you're expected to get the job done.\nEvery application requires constant bug fixing, refactoring and sometimes even monkey patching. Maintaining code is no amusement park; especially if it's not your own.\n\n \n \n \n \nBeing a Junior Developer\nFresh out of school you might think you're a pretty kick-ass programmer. Let me tell you: you're not.\nYou wouldn't guess what talented people can do with these blinking machines.\nYou'll have tons of things to learn in the first few years.\nProfessional software development is a lengthy process. Writing readable, well-tested, well-documented code is a substantial effort. You will need patience, lots of it. Both, with yourself and with others.\nAs a junior, you only think in black and white. You look at some code, and it's all wrong. Who in their right mind created this horrible monstrosity?!\nAs you become more experienced, you'll see the shades of grey.\nEventually, you'll understand that those neckbeards were not slower than you, but\nmore careful. You learn how to test your code, how to document it. You even begin to\nappreciate UML diagrams.\n\n \n \n \n \nBecoming obsolete\n"The world is moving too fast. What you learned today is obsolete tomorrow. Why bother?".\nI've heard that saying countless times throughout my career.\nIt's both, popular and wrong.\nIf a skill becomes obsolete, it's not a skill.\nThroughout your career you don't want to be known as "the Jenkins guy", you want to be the\nexpert in Software Quality. Hint: If you don't know what Jenkins is, that's the\nwhole point. You should not narrow down your scope too much.\nThe right skills never become obsolete.\nFrom time to time it happens, that due to some new company policy your beautiful creation will become obsolete.\nAs depressing as it sounds: it's a regular part of the software business.\nYou need to adapt.\nOne advice I can give you is not to take it too seriously.\nDrop the project, keep the wisdom.\nEmbrace change.\n\n \n \n \n \nWriting software in a non-perfect world\nA professional programmer has to deal with deficiencies all the time. The game is called "balancing constraints". Deadlines, budgets, and code quality are just a few competing constraints we have to consider.\nElegant designs fade away in the face of reality.\nIn the end you want to earn money with your software, so you have to ship it!\nThe best developers I know, keep the balance between pragmatism and elegance.\nThey know which parts matter and which don't. Those who don't will be replaced\nwhen there's a need.\nFor me, I was always leaning more towards elegance.\nThat's just a nicer way to say I was a perfectionist.\nI needed to learn the pragmatic part through hard work.\n\n \n \n \n \nMentoring less experienced Programmers\n\nThe better you become at programming, the less you code.\n\nInstead, you will spend more time thinking about Software Architecture,\nhigh-level designs and splitting up the work into smaller junks for other developers to consume.\nYou will start mentoring Junior Developers. Recruiting will require a lot of your\nattention. You will spend your time in Meetings, discussing project goals with\nbusiness people.\nOne might say, you take the role of a mediator. Others might call you a manager.\nOnce you know the ins and outs of the business, you are an essential asset for\nthe company. You might get asked to become a manager, or at least managing projects will slowly feel like a natural extension of your responsibilities.\nBut beware! This slow and gradual process is dangerous.\nMoving back to being a full-time programmer is not easy.\nDuring the time you were busy with project management, others were busy improving their\ncoding skills.\nYou can try to keep up-to-date in your free time but that's hard.\nI've seen excellent developers become great managers. At some point in your career\nit's a decision you need to make for yourself.\nHowever you decide, it pays off to invest some time into learning how to\ncommunicate. Empathy plays a prominent role in that.\nDeveloping software as a team is so complicated that a lot of time is spent on aligning goals and communicating problems. In fact, communication is what you get paid for. This includes documentation, tests and the code itself.\nTalk to others, listen to their problems. Read books about Software Project\nManagement, even though you don't want to be a manager yourself. It will help\nyou understand the role of your boss.\n\n \n \n \n \nA word about money\nThere are many good reasons to work in IT, but money is not one of them.\nWhile it can be tempting to base your career decisions on prospective salary,\ndon't do it. You will be very unhappy. You will spend eight hours or more each day sitting in front of a blinking cursor.\nThat's a lot of time, and time is much more valuable than money.\nDon't get me wrong. There's plenty of jobs that pay well.\nYou will most likely not get rich, though. If you want\nto make it big, I can't help you. Maybe look into Real Estate or so...\nThe only way to get rich as a developer is to work on something really hard, put in lots of hours and get\nlucky. Startups, basically. Keep in mind: One Bill Gates takes a thousand failed\nattempts.\nAnother way is to stop being a programmer and become a manager instead.\nI've already shared my opinion on that in the last section.\n\n \n \n \n \nFinal words\nWhile you should learn to read (and maybe write) code, working as a professional programmer is not for everyone.\nYou might ask: "Is it worth it?".\nFor me it was the right decision. Hopefully I could help you to make your own.\n" }, { "title": "The Future of Rust", "url": "https://endler.dev/2017/future-of-rust/", "body": "Let me first point out the obvious: yes, the title is a little sensationalist. Also\nyou might be asking why I should be entitled to talk about the future of Rust. After\nall, I'm neither part of the Rust core team, nor a major contributor to the Rust\necosystem. To that I answer: why not? It's fun to think about the future of\nsystems programming in general and Rust in particular.\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n Ferris is the inofficial Rust mascot\n \n \n \n \n \n Source: Illustration provided by zooenvato for FreePik.com\n \n \n \n\nYou might have heard of the near-term goals that the core team has committed itself to. Faster compile times and a more gentle learning curve come to mind.\nThis post is not about that.\nInstead, I want to explore some more exotic areas where Rust could shine in\nfive to ten years from now. To make it big, we need both, roots and wings.\n\n \n \n \n \nData Science\nRight now, the most popular languages for Data Science are Python, Java, R, and C++.\n\n \n Programming language popularity for data science (Source).\n \n\nWe've observed that while prototypes are mostly written in dynamically typed\nlanguages like Python and R, once an algorithm reaches production level quality\nit is often rewritten in faster languages such as C++ for scalability.\nIt is not unthinkable that Rust is going to be some healthy competition for C++ in the near future.\nThe benchmarks of leaf, a machine learning library written in Rust, are already nothing short of\nimpressive.\n\n \n \n \n \nBlockbuster games\nGames are another area where Rust might shine.\nIt's financially attractive for Game Studios to support multiple platforms without much\neffort. Cargo and rustup make cross-compiling easy.\nModern libraries slowly fill the tooling gaps for large-scale game development.\nRust's support for the Vulkan 3D graphics API might already be the best of class.\nThe killer feature though is the unique combination of safety and performance.\nIf you ship a game to a million players and they throw money at you, you'll better make sure that it doesn't crash... right?\nThat said, the first AAA Rust game might still be far in the future. Here's Blizzard's standpoint on Rust in 2017.\n\n \n \n \n \nSystems Engineering\nMaybe β€” eventually β€” we will also see formal verification of the Rust core. Projects like RustBelt would then open new opportunities in safety-focused industries like the Space industry. Wouldn't it be nice to safely land a Spacecraft on Mars that is controlled by Rust? (Or by one of its spiritual successors.)\nI wonder if SpaceX is experimenting with Rust already...\n\n \n \n \n \nIntegrating with other languages\nThere are many other areas I haven't even mentioned yet. For example, financial and medical software or Scientific Computing, just to name a few.\nIn all cases, Rust might be a good fit. Right now the biggest barrier to entry\nis probably the huge amount of legacy code. Many industries maintain large codebases in Cobol,\nC or Fortran that are not easily rewritten.\nFortunately, Rust has been proven to work very nicely with other languages.\nPartly because of strong C-compatibility and partly because there is no Runtime or Garbage Collector.\nA typical pattern is to optimize some core part of an application in Rust that has hard safety/performance\nrequirements, while leaving the rest untouched.\nI think this symbiosis will only become stronger in the long run.\nThere are even ambitious projects like Corrode which attempt to translate C code to Rust automatically.\n\n \n \n \n \nSummary\nOverall I see huge potential for Rust in areas where safety, performance or total control over the machine are essential. With languages like Rust and Crystal, a whole class of errors is a thing of the past. No null pointers, no segmentation faults, no memory leaks, no data races.\nI find it encouraging that future generations of programmers will take all that for granted.\n" }, { "title": "Launching a URL Shortener in Rust using Rocket", "url": "https://endler.dev/2017/rust-url-shortener/", "body": "One common systems design task in interviews is to sketch the software architecture of a URL shortener (a bit.ly clone, if you may).\nSince I was playing around with Rocket – a web framework for Rust – why not give it a try?\n\n \n\n \n \n \n\n \n\n \n \n\n \n \n \n \n A rocket travelling through space\n \n \n \n \n \n\n\n \n \n \n \nRequirements\nA URL shortener has two main responsibilities:\n\nCreate a short URL for a longer one (d'oh!).\nRedirect to the longer link when the short link is requested.\n\nLet's call our service rust.ly (Hint, hint: the domain is still available at the time of writing...).\nFirst, let's create a new Rust project:\n\ncargo new --bin rustly\n\nNext, we add Rocket to our Cargo.toml:\n\n[dependencies]\nrocket = "0.2.4"\nrocket_codegen = "0.2.4"\n\nWarning: Most likely you need to get the very newest Rocket version.\nOtherwise, you might get some... entertaining error messages. Find the newest\nversion on crates.io.\nSince Rocket requires cutting-edge Rust features, we need to use a recent nightly\nbuild. Rustup provides a simple way to switch between stable and nightly.\n\n πŸ€” Nightly Rust might no longer be required. Has anyone tried without and can\nreport back?\n\n\nrustup update && rustup override set nightly\n\n \n \n \n \nA first prototype\nNow we can start coding our little service.\nFirst, let's write a simple "hello world" skeleton to get started.\nPut this into src/main.rs:\n\n#![feature(plugin)]\n#![plugin(rocket_codegen)]\n\nextern crate rocket;\n\n#[get("/<id>")]\nfn lookup(id: &str) -> String {\n format!("⏩ You requested {}. Wonderful!", id)\n}\n\n#[get("/<url>")]\nfn shorten(url: &str) -> String {\n format!("πŸ’Ύ You shortened {}. Magnificent!", url)\n}\n\nfn main() {\n rocket::ignite().mount("/", routes![lookup])\n .mount("/shorten", routes![shorten])\n .launch();\n}\n\nUnder the hood, Rocket is doing some magic to enable this nice syntax.\nMore specifically, we use the rocket_codegen crate for that.\nIn order to bring the rocket library into scope, we write extern crate rocket;.\nWe defined the two routes for our service. Both routes will respond to a GET request.\nThis is done by adding an attribute named get to a function.\nThe attribute can take additional arguments.\nIn our case, we define an id variable for the lookup endpoint and a url variable for the shorten endpoint.\nBoth variables are Unicode string slices. Since Rust has awesome Unicode support, we respond with a nice emoji just to show off. πŸ•Ά\nLastly, we need a main function, which launches Rocket and mounts our two routes. This way, they become publicly available.\nIf you want to know even more about the in-depth details, I may refer you to the official Rocket documentation.\nLet's check if we're on the right track by running the application.\n\ncargo run\n\nAfter some compiling, you should get some lovely startup output from Rocket:\n\nπŸ”§ Configured for development.\n => address: localhost\n => port: 8000\n => log: normal\n => workers: 8\nπŸ›° Mounting '/':\n => GET /<hash>\nπŸ›° Mounting '/shorten':\n => GET /shorten/<url>\nπŸš€ Rocket has launched from https://localhost:8000...\n\nSweet! Let's call our service.\n\n> curl localhost:8000/shorten/www.endler.dev\nπŸ’Ύ You shortened www.endler.dev. Magnificent!\n\n> curl localhost:8000/www.endler.dev\n⏩ You requested www.endler.dev. Wonderful!\n\nSo far so good.\n\n \n \n \n \nData storage and lookup\nWe need to keep the shortened URLs over many requests... but how?\nIn a production scenario, we could use some NoSQL data store like Redis for that.\nSince the goal is to play with Rocket and learn some Rust, we will simply use an\nin-memory store.\nRocket has a that feature called managed state.\nIn our case, we want to manage a repository of URLs.\nFirst, let's create a file named src/repository.rs:\n\nuse std::collections::HashMap;\nuse shortener::Shortener;\n\npub struct Repository {\n urls: HashMap<String, String>,\n shortener: Shortener,\n}\n\nimpl Repository {\n pub fn new() -> Repository {\n Repository {\n urls: HashMap::new(),\n shortener: Shortener::new(),\n }\n }\n\n pub fn store(&mut self, url: &str) -> String {\n let id = self.shortener.next_id();\n self.urls.insert(id.to_string(), url.to_string());\n id\n }\n\n pub fn lookup(&self, id: &str) -> Option<&String> {\n self.urls.get(id)\n }\n}\n\nWithin this module we first import the HashMap implementation from the standard library.\nWe also include shortener::Shortener;, which helps us shorten the URLs in the next step. Don't worry too much about that for now.\nBy convention, we implement a new() method to create a Repository struct with an empty HashMap and a new Shortener. Additionally, we have two methods, store and lookup.\nstore takes a URL and writes it to our in-memory HashMap storage. It uses our yet-to-be-defined shortener to create a unique id. It returns the shortened ID for the entry.\nlookup gets a given ID from the storage, and returns it as an Option. If the ID is found, the return value will be Some(url); if there is no match it will return None.\nNote that we convert the string slices (&str) to String using the to_string() method. This way we don't need to deal with lifetimes. As a beginner, don't think too hard about them.\n\n \n \n \n \nAdditional remarks (can safely be skipped)\nA seasoned (Rust) developerβ„’ might do a few things differently here. Did you notice the tight coupling between the repository and the shortener? In a production system, Repository and Shortener might simply be concrete implementations of traits (which are a bit like interfaces in other languages, but more powerful). For example, Repository could implement a Cache trait:\n\ntrait Cache {\n // Store an entry and return an ID\n fn store(&mut self, data: &str) -> String;\n // Look up a previously stored entry\n fn lookup(&self, id: &str) -> Option<&String>;\n}\n\nThis way we get clear sepration of concerns, and we can easily switch to a different implementation (e.g. a RedisCache). Also, we could have a MockRepository to simplify testing. Same for Shortener.\nOn top of that, you might want to use the Into trait to support both, &str and String as parameters of store:\n\npub fn store<T: Into<String>>(&mut self, url: T) -> String {\n\t\tlet id = self.shortener.shorten(url);\n\t\tself.urls.insert(id.to_owned(), url.into());\n\t\tid\n}\n\nIf you're curious about this, read this article from Herman J. Radtke III.\nFor now, let's keep it simple.\n\n \n \n \n \nActually shortening URLs\nLet's implement the URL shortener itself.\nYou might be surprised how much was written about URL shortening all over the web.\nOne common way is to create short URLs using base 62 conversion.\nAfter looking around some more, I found this sweet little crate called harsh, which perfectly fits the bill. It creates a hash id from an input string.\nTo use harsh, we add it to the dependency section of our Cargo.toml:\n\nharsh = "0.1.2"\n\nNext, we add the crate to the top of to our main.rs:\n\nextern crate harsh;\n\nLet's create a new file named src/shortener.rs and write the following:\n\nuse harsh::{Harsh, HarshBuilder};\n\npub struct Shortener {\n id: u64,\n generator: Harsh,\n}\n\nimpl Shortener {\n pub fn new() -> Shortener {\n let harsh = HarshBuilder::new().init().unwrap();\n Shortener {\n id: 0,\n generator: harsh,\n }\n }\n\n pub fn next_id(&mut self) -> String {\n let hashed = self.generator.encode(&[self.id]).unwrap();\n self.id += 1;\n hashed\n }\n}\n\nWith use harsh::{Harsh, HarshBuilder}; we bring the required structs into scope. Then we define our own Shortener struct, which wraps Harsh. It has two fields: id stores the next id for shortening. (Since there won't be any negative ids, we use an unsigned integer for that.) The other field is the generator itself, for which we use Harsh.\nUsing the HarshBuilder you can do a lot of fancy stuff, like setting a custom alphabet for the ids. We're good for now, but for more info, check out the official docs.\nWith next_id we retrieve a new String id for our URLs.\nAs you can see, we don't pass the URL to next_id. That means we actually don't shorten anything. We merely create a short, unique ID. That's because most hashing algorithms produce fairly long URLs and having short URLs is kind of the whole idea.\n\n \n \n \n \nWiring it up\nSo we are done with our shortener and the repository.\nWe need to adjust our src/main.rs again to make use of the two.\nThis is the point where it gets a little hairy.\nI have to admit that I struggled a bit here.\nMainly because I was not used to multi-threaded request handling. In Python or\nPHP you don't need to think about shared-mutable access.\nInitially I had the following code in my main.rs:\n\n#[get("/<url>")]\nfn store(repo: State<Repository>, url: &str) {\n repo.store(url);\n}\n\nfn main() {\n rocket::ignite().manage(Repository::new())\n .mount("/store", routes![store])\n .launch();\n}\n\nState is the built-in way to save data across requests in Rocket. Just tell it what belongs to your application state with manage() and Rocket will automatically inject it into the routes.\nBut the compiler said no:\n\nerror: cannot borrow immutable borrowed content as mutable\n --> src/main.rs\n |\n | repo.store(url);\n | ^^^^ cannot borrow as mutable\n\nIn hindsight it all makes sense: What would happen if two requests wanted to modify our repository at the same time?\nRust prevented a race condition here! Yikes.\nAdmittedly, the error message could have been a bit more user-friendly, though.\nFortunately, Sergio Benitez (the creator of Rocket) helped me out on the Rocket IRC channel (thanks again!).\nThe solution was to put the repository behind a Mutex.\nHere is our src/main.rs in its full glory:\n\n#![feature(plugin, custom_derive)]\n#![plugin(rocket_codegen)]\n\nextern crate rocket;\nextern crate harsh;\n\nuse std::sync::RwLock;\nuse rocket::State;\nuse rocket::request::Form;\nuse rocket::response::Redirect;\n\nmod repository;\nmod shortener;\nuse repository::Repository;\n\n#[derive(FromForm)]\nstruct Url {\n url: String,\n}\n\n#[get("/<id>")]\nfn lookup(repo: State<RwLock<Repository>>, id: &str) -> Result<Redirect, &'static str> {\n match repo.read().unwrap().lookup(id) {\n Some(url) => Ok(Redirect::permanent(url)),\n _ => Err("Requested ID was not found.")\n }\n}\n\n#[post("/", data = "<url_form>")]\nfn shorten(repo: State<RwLock<Repository>>, url_form: Form<Url>) -> Result<String, String> {\n let ref url = url_form.get().url;\n let mut repo = repo.write().unwrap();\n let id = repo.store(&url);\n Ok(id.to_string())\n}\n\nfn main() {\n rocket::ignite().manage(RwLock::new(Repository::new()))\n .mount("/", routes![lookup, shorten])\n .launch();\n}\n\nAs you can see we're using a std::sync::RwLock here, to protect our repository from shared mutable access. This type of lock allows any number of readers or at most one writer at the same time.\nIt makes our code a bit harder to read because whenever we want to access our repository, we need to call the read and write methods first.\nIn our lookup method, you can see that we are returning a Result type now. It has two cases: if we find an id in our repository, we return Ok(Redirect::permanent(url)), which will take care of the redirect. If we can't find the id, we return an Error.\nIn our shorten method, we switched from a get to a post request.\nThe advantage is, that we don't need to deal with URL encoding. We just create a struct Url and derive FromForm for it, which will handle the deserialization for us. Fancy!\nWe're done. Let's fire up the service again and try it out!\n\ncargo run\n\nIn a new window, we can now store our first URL:\n\ncurl --data "url=https://www.endler.dev" https://localhost:8000/\n\nWe get some ID back that we can use to retrieve the URL again. In my case, this was gY.\nPoint your browser to https://localhost:8000/gY and you should be redirected to my homepage.\n\n \n \n \n \nSummary\nRocket provides fantastic documentation and a great community.\nIt really feels like an idiomatic Rustlang web framework.\nI hope you had some fun while playing with Rocket.\nYou can find the full example code on Github.\n" }, { "title": "The Essence of Information", "url": "https://endler.dev/2017/the-essence-of-information/", "body": "People look confused when I tell them about my passion for algorithms and data-structures.\nMost of them understand what a Programmer is doing, but not what Computer Science is good for.\nAnd even if they do, they think it has no practical relevance.\nLet me show you with a simple example, that applied Computer Science can be found everywhere.\nImagine a pile of socks that need to get sorted.\nNot exactly the most exciting pastime.\nYou've put off this task for so long, that it will inevitably take an hour to be done.\n\n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n Yes, there is a game about sorting socks.\n \n \n \n \n \n Source: It's called Sort the Socks and you can get it for free on the App Store.\n \n \n \n\nConsidering your options, you decide to get some help.\nTogether with a friend you get to work. You finish in roughly half the time.\nA Computer Scientist might call this pile of socks a resource.\nYou and your friend get bluntly degraded to workers.\nBoth of you can work on the problem at the same time β€” or in parallel.\nThis is the gist of Parallel Computing.\nNow, some properties make sock-sorting a good fit for doing in parallel.\n\nThe work can be nicely split up. It takes about the same time for every worker to find a pair of socks.\nFinding a different pair is a completely separate task that can happen at the same time.\n\nThe more workers you assign to this task, the faster you're done.\n\n1 worker takes 60 minutes.\n2 workers take 30 minutes.\n\nHow long will 3 workers take? Right! Around 20 minutes. We could write down\na simple formula for this relationship:\n\nWell, that is not quite correct. We forgot to consider the overhead: When Mary\ntries to pick up a sock, Stephen might reach for the same.\nThey both smile and one of them picks another sock.\nIn computing, a worker might do the same. Well, not smiling but picking another\ntask. When lots of workers share resources, these situations occur quite\nfrequently. And resolving the situation always takes a little extra time. So we are a\nbit away from our optimal sorting speed because of that.\nBut it gets worse! Let's say you have 100 workers for 100 socks.\nIn the beginning, every worker might take one sock and try to find a match for\nit. Here's the problem: As soon as they pick up one sock each, there are no\nsocks left. All workers are in a waiting state. The sorting takes forever.\nThat's a deadlock, and it's one of the most frightening scenarios of parallel computing.\nIn this case, a simple solution is to put down the sock again and wait for some time until trying to get a new sock.\nAnother way out of the dilemma would be, to enforce some kind of "protocol" for sorting.\nThink of a protocol as a silent agreement between the workers on how to achieve a common goal.\nSo, in our case, each worker might only be responsible for one color of socks.\nWorker one takes the green socks, worker two the gray ones and so on.\nWith this simple trick, we can avoid a deadlock, because we work on completely\nseparate tasks.\nBut there's still a catch. What if there are only four green socks and 4000 gray socks?\nWorker one would get bored fairly quickly. He would sort the two pairs of socks in\nno time and then watch worker two sort the rest.\nThat's not really team spirit, is it?\nSplitting up the work like this makes most sense, if we can assume that we\nhave around the same number of socks for every color.\nThis way we achieve roughly the same workload for\neveryone.\nThe following histogram gives you an idea of what I mean:\n\nIn this case, we have about equally sized piles for each color. Looks\nlike a fair workload for every worker to me.\n\nIn the second case, we don't have an equal distribution. I don't want to sort the\ngray socks in this example. We need to think a little harder here.\nWhat can we do?\nMost of the time it helps to think of other ways to split up work.\nFor example, we could have two workers sort the big gray pile together. One\nsorts the large socks; the other one sorts the small ones. We run into another problem, though: Who decides what "large" and "small" means in this case?\nSo, instead of thinking too hard about a smarter approach, we decide to be\npragmatic here. Everyone just grabs an equally sized pile of socks β€” no\nmatter the color or the size β€” and gets\nto work.\nMost likely, there will be some remaining socks in each pile, which have no match.\nThat's fine. We just throw them all together, mix the socks, create new piles from\nthat, and sort them again. We do so until we're done.\nWe call that a task queue. It has two advantages: First, you don't need any additional agreements between the workers and second, it scales reasonably\nwell with the number of workers without thinking too hard about the problem\ndomain.\nThe tricky part about distributed systems is, that seemingly straightforward solutions can fail\nmiserably in practice.\nWhat if our small piles look like this?\n\nThe number of pairs in each pile is... sobering.\nWhat we could do is run a very quick presorting step to increase the number of matches. Or maybe you come up with an even better idea?\nThe cool thing is, once you have found a faster approach, it works for similar tasks, too.\nProblems like this have their roots in Computer Science, and they can be found everywhere.\nPersonally, I don't like the term Computer Science too much. I prefer\nthe German term "Informatik", which I would roughly translate as "Information Science".\nBecause the real essence of what we're doing here is to find a general way to solve a\nwhole class of problems. We think of the nature of objects and their properties.\nWe don't sort socks; we try to answer the fundamental questions of information. Maybe now you can understand why I'm so passionate about this subject.\nOh, and here's a related post about why I love programming.\n" }, { "title": "Why I Love Programming", "url": "https://endler.dev/2017/why-i-love-programming/", "body": "Programming has many faces. It is the science of structured thinking.\nIt is the art of eloquent expression.\nIt teaches you to be humble when you look at other peoples' fascinating work.\nMost of all, it teaches you a lot about yourself.\nWhile the syntax may change, the concepts will not.\n\nThis post is split into two parts.\nIn the first part, I will talk about the joy of programming.\nThe second part will deal with the notion of being a professional programmer.\nIf you're not sure yet whether you want to learn how to program, this article is for you.\n\n \n \n \n \nAutomating stuff gives you superhero strengths\nBeing able to program is infinitely rewarding. You can help your sister sort a\nthousand pictures in a few seconds. You write a little backup\nscript for your grandma. The possibilities are endless.\n\n \n \n \n \nCoding is fun!\nCoding something is more fun than using it. It's even better than playing games.\nWhy? Learn how to program a computer and get the best games for free β€” your own.\nYou're in total control. It's your idea, your logic, even your laws of physics.\nIt's like building a house but without paying anything for the\nbuilding materials. You can build a mansion for free.\n\n \n \n \n \nSharing is fun, too!\nTo get new inspiration for your next project, read the programs of others.\nThis will give you an idea of how they think and how they solve problems.\nMany great programmers share their best code with you.\nYou can do the same and share your project - or just the prettiest parts of it - with other programmers.\nWatching somebody else use your work is one\nof the most satisfying things you will ever experience.\nIt's very fulfilling to see your tool serve a purpose it wasn't built for.\n\n \n \n \n \nElegant, creative solutions\nIt's very appealing to work so hard on your vision that everything unnecessary peels off.\nAll these little ideas and fundamental insights suddenly fall into place.\nWhat's remaining is the distilled truth, the result of an ambitious but rewarding thought process\nand when you write it down as a program you can see all the little pieces working together.\nThis makes it so gratifying to figure stuff out on your own.\nProgramming is about understanding a problem so thoroughly, that you can teach a\npiece of metal how to solve it.\nEven the way your program is structured can be a piece of art.\nIt can be concise, witty and fast all at the same time.\n\n \n \n \n \nTalk to a machine\nIt's fascinating that something is understood by machines and humans using the same language.\nI'm baffled when I realize that these circuits can actually "understand" and interpret words - in a way.\n\n \n \n \n \nStanding on the shoulders of giants\nTalking to other programmers and watching them work is a fascinating inspiration.\nThe very system you are using to read this text relies on their work.\nEven if you're far apart, you can study their work on Open Source projects online.\nBut if you get a chance, watch them giving talks at conferences and meet them at local user groups.\nBecoming part of a community is gratifying.\nTo exchange ideas and to collaborate on projects helps you push your boundaries and learn something new every day.\n\n \n \n \n \nHave fun, forget the rest\nThe machine is agnostic to your skin color. It doesn't matter if you're a twelve-year-old girl or a lecturer at University.\nIf you keep making the same mistake for ten hours straight, your computer won't scream at you. It won't punish you. It will happily await your commands. Also, the hurdles of entry are pretty low. An old computer is enough; even pen and paper and a book will suffice to work on cool programming ideas.\n\n \n \n \n \nGet started!\nYou choose your own projects; nobody else.\nDon't let anybody tell you that you're not smart enough for this stuff. Ever.\nEach program is a wonderful journey so join us and code the world around you.\n" }, { "title": "Tools", "url": "https://endler.dev/2011/tools/", "body": "For as long as I can think, religious flamewars have infected computer science.\nHaving arguments about technical topics can be healthy, but flamewars are not. I'm sick of it.\nI'm fed up with people telling me that their work environment is oh-so better,\nfaster and so on. That's fine, but it doesn't matter. Your equipment only plays a supporting role. You don't even need\na computer to do programming. Donald Knuth wrote algorithms on a\nnotepad. Alan Turing wrote the first chess computer on a piece of\npaper. And it worked. Beat that!\nFor an average user, the next best system is probably good enough. Just a few bucks and you get an excellent piece of hardware which is completely sufficient to surf the web, chat, archive photos, write documents, listen to music and watch movies. You can do that with a Pentium IV, 256 MB RAM and any recent Operating System (you will likely get that one for free). Heck, you can use your old Commodore for most of that. Computers have been mature and reliable enough to do all that for ages. There's no need to upgrade your system for Farmville, just like there's no reason to buy a new car if the old one works perfectly fine. When it comes to software, many of us still use Office 2000 or Photoshop 8 or VisiCalc without feeling the urge to upgrade.\nProfessionals find themselves in a similar situation. Well, maybe we invest a bit more money, but still, our hardware is incredibly cheap compared to our salary (hopefully). Nothing is perfect, but most of the time it's good enough. That compiler you were using a decade ago? Still does the job. We are still using slightly modified descendants of programming languages from computing stone-age. Even if you're doing numerical computing for NASA, your primary work environment is a black box running a text editor or an IDE.\nI don't care what you are using to get things done. Find an environment that suits your needs and be happy with it. Maybe you use Emacs on a Lemote Yeelong netbook (hello Richard Stallman) or Vim on your workstation. It's the same thing: A text editor running on a piece of metal.\nYou're not a worse programmer for using Nano, ed or TextMate. Notepad works just fine, too. It loads files, saves files and lets you edit them in between. That's a hell lot more functionality than Bill Gates and Paul Allen had when they wrote a BASIC interpreter for the Altair. If you find something you're happy with, just stick with it but don't start arguing. It isn't worth your time.\nDon't feed the trolls. When it comes to software, don't fall into the old FreeBSD vs. Linux vs. Windows vs. mum cliche. Instead, talk about your code. Let's look at your problem-solving skills. Let's be pragmatic here.\n\nTalk is cheap. Show me the code. - Linus Torvalds\n\nI don't care which programming language you are using. Java? Fine. Visual Basic? Great! Scala, Cobol, PHP, C++? All fine. Write in Assembler or lolcode. Don't moan about the fact that language X is missing feature Y. Write a library or use something different. Stop saying JavaScript is a toy language. It just doesn't fit your needs. Instead, show me your Lisp adventure game. Write an interpreter for Brainfuck. Do something. Move things.\nConcerning PHP, nir wrote on Hacker News:\n\nAny idiot can write a snarky comment about PHP. Very few get to write code that has anywhere near the impact it had.\n\nWill you fall off your chair when I admit that I like the PHP syntax?\nOK, it has its rough edges (do we really need the $ sign?) but what's\nmore important is how much I can get done with it. PHP was my long time\ngo-to language for off the hook, one time scripts. It looks a bit ugly\nbut it runs on any server and comes with an enormous amount of built-in\nfunctionality. It's great for rapid prototyping and gluing things together.\nIn fact, when you write a piece of software, what you should strive for is to produce quite good software and what you really need to accomplish is good enough software to make your users happy.\nZed A. Shaw puts it quite nicely in the afterword to Learn Python the hard way\n\nI have been programming for a very long time. So long that it is incredibly boring to me. At the time that I wrote this book I knew about 20 programming languages and could learn new ones in about a day to a week depending on how weird they were. Eventually though this just became boring and couldn't hold my interest. What I discovered after this journey of learning was that the languages didn't matter, it was what you did with them. Actually, I always knew that, but I'd get distracted by the languages and forget it periodically. The programming language you learn and use does not matter. Do not get sucked into the religion surrounding programming languages as that will only blind you to their real purpose of being your tool for doing interesting things.\n\nDon't get emotional for any tool you use. An iPhone - I'm sorry to disappoint you - is just a phone. No magic. No "think different". "But it's evil!", the ether says, "it's not open source". Well, Android just exists because Google needed to rapidly develop a mobile platform. It's simply part of their business. There is no moral behind that. Google is a yet another company just like Microsoft or Apple.\nMy MacBook serves me as a solid tool, but if something "better" comes around, I will happily kick it out. I've ditched Firefox after five years just because Chrome is faster and I will get rid of Chrome when I find a worthy successor.\nVim is quite good in my opinion but if there's a faster way to do things I'm not afraid to dump it. Instead get your hands dirty and fix the problems or craft something new.\n" }, { "title": "Are you a Programmer?", "url": "https://endler.dev/2011/are-you-a-programmer/", "body": "My geography teacher once told the story of her first lecture at University.\nAs an introduction, her professor asked the class to draw\na map of Germany without any help and as accurate as possible. To her surprise, she was not\nable to fill the map with much detail. Even the shape of the country was a bit vague.\nShe had seen thousands of images of Germany (her mother country) but\nwasn't able to reproduce it from her blurry memory. She would have to look it up.\nDoesn't this sound familiar? We rely on machines to manage large portions\nof our knowledge. There's hard work involved to learn something by heart.\nHere is a similar test for programmers:\n\nUsing a programming language of your choice, write a correct sorting\nalgorithm with an average runtime complexity of O(n*log n) (Heapsort,\nQuicksort, Bucketsort, you name it) on a piece of paper without the help of any\nexternal tools.\n\nAnd by correct I mean it must be free of bugs without any modifications when you type it in.\nYou would be surprised by the large percentage of professional software\nengineers who can't pull this off.\nSome might argue that knowledge about details of programming language\nsyntax is unimportant: "Why learn all the little nitpicks when you know\nhow to use a search engine? Why start with a clean slate when you can easily\ncopy, paste and modify an example from a tutorial?\nEvery few years/months I have to completely relearn the syntax for a different language anyway."\nBut that is a myth. If you know only\none programming language really well - even if it is something\noutdated like Fortran or COBOL - you could easily earn a fortune with\nthat knowledge. Suppose you started with C in 1975. You could still\nuse the same syntax today - almost four decades later.\nSame for text editors. Emacs and Vim are both decades\nold. They are battle-hardened. I don't care which one you prefer, but you\nwill spend a large part of your life with your tools so invest the time to master them.\nAs a side note, it appears that very few people strive for perfection in anything they do.\nThey happily settle for "good enough". This can have many different reasons, and I'm not\nblaming anybody for not doing his homework but maybe I'm not alone with\nthat observation.\nIf you don't know how to use your tools without a manual, you are a lousy craftsman.\nIf you need a dictionary to write a simple letter, you will have a hard\ntime becoming a writer because it would already be challenging for you to form elegant, fluent\nsentences β€” let alone engaging and original stories.\nI don't want to read these books.\nWhat makes a programmer?\n\nShe has at least one programming language she knows inside out.\nShe can implement standard algorithms (i.e. for sorting, searching)\nand data-structures (i.e. trees, linked lists) which are robust and\nreasonably fast on the fly.\nShe has at least a basic understanding of complexity theory and\nprogramming concepts like recursion and pointers.\n\nBut, to be a good programmer, you should\n\nBe able to code in at least two fundamentally different programming\nparadigms (i.e. declarative, functional).\nHave experience with big software architectures.\nBe familiar with your programming environment like the operating system and a sophisticated text editor of your choice. Preferably one, that is\neasily extendable.\n\nAnd that is just the tip of the iceberg.\n"There's too much to learn!", I hear some of you say.\nStart slowly.\nYou need only three commands to start with Vim: i, ESC, :wq.\nThat's enough for day one.\nI realize that most of these essentials won't be taught during lectures.\nYou have to learn a vast portion on your own.\nBut let's face it: If you don't know this stuff, you are not a programmer, you're a freshman.\n" }, { "title": "On Hard Work", "url": "https://endler.dev/2011/on-hard-work/", "body": "Great people get shaped by their achievements\n\nThere's Thomas Edison who developed countless prototypes before selling a single light bulb.\nThe unemployed Joanne K. Rowling writing Harry Potter in a Cafe while caring for her child.\nSteve Wozniak creating the first personal computer in his spare time while working at HP.\n\nWhat do they have in common?\nThey all lived through frustration and contempt but still reached their goals, even though the chances for success were\nlow. These people are stemming their strong will from an intrinsic curiosity.\n\n \n \n \n \nDedication\nSure, I love what I do. I want to be a programmer for the rest of my life, but sometimes it seems simply too hard to finish a project.\nI get scared by the big picture and fear that I won't finish on time. What I need is a different mindset.\nDhanji R. Prasanna, a former Google Wave team member made this observation\n\nAnd this is the essential broader point--as a programmer you must have a series of wins, every single day. It is the Deus Ex Machina of hacker success. It is what makes you eager for the next feature, and the next after that.\n\nWhile Google Wave has not been commercially successful, it sure was a\ntechnical breakthrough β€” and it was a drag to push it out into public.\nWe always have to see our goal right in front of us, as we take a billion baby steps to reach it.\nThis is true for any profession. Winners never give up.\n\n \n \n \n \nDirection\nToday it is easier to accomplish something meaningful than ever before.\nIf you are reading this, you have access to a powerful instrument β€” a\ncomputer with an Internet connection. We live in a time where a single\nperson can accomplish miracles without hard physical labor.\nA time where billions of people can grow a business from their desk, get famous in minutes,\npublish books in seconds and have instant access to large amounts of\ndata. The most potent development over the last 100\nyears has been the reduction of communication costs. Transferring a bit of\ninformation to the other end of the world is virtually free and takes\nfractions of a second. While proper education was a privilege of a lucky few\nwell into the 20th century, learning new things is now mostly a question of\nwill.\nNevertheless, learning is still a tedious task,\nrequiring patience and determination.\nAs the amount of information has increased, so have the ways of distraction.\nLosing focus is just a click away.\n\n \n \n \n \nDevotion\nEverybody can start something. Few will finish anything.\nThat's because getting things done is hard, even if you love what\nyou're doing. (Watch the beginnings of There Will Be Blood and Primer for a\ndefinition of hard work.)\nNo matter what they tell you, achieving anything sustainable means hustling. It means making\nsacrifices. It means pushing through.\nIt means selling something even though it isn't perfect. Your beautiful project might turn into an ugly groundhog in\nthe end. Put makeup on it and get it out the door.\nOn a report about Quake's 3D-Engine, developer Michael Abrash says:\n\nBy the end of a project, the design is carved in stone, and most of the work involves fixing bugs, or trying to figure out how to shoehorn in yet another feature that was never planned for in the original design. All that is a lot less fun than starting a project, and often very hard work--but it has to be done before the project can ship. As a former manager of mine liked to say, "After you finish the first 90% of a project, you have to finish the other 90%." It's that second 90% that's the key to success.\n\n\nA lot of programmers get to that second 90%, get tired and bored and frustrated, and change jobs, or lose focus, or find excuses to procrastinate. There are a million ways not to finish a project, but there's only one way to finish: Put your head down and grind it out until it's done. Do that, and I promise you the programming world will be yours.\n\nThat last part has influenced me a lot.\nThe dedication, the urgency to reach your aims must come from within you.\nIt's your raw inner voice speaking β€” don't let it fade away.\nAnd when you are close to giving up, stop thinking so hard. Just try to\npush forward and make a tiny step in the right direction.\nShip it!\n" }, { "title": "Overkill – Java as a First Programming Language", "url": "https://endler.dev/2010/overkill-java-as-a-first-programming-language/", "body": "I recently talked to a student in my neighborhood about his first programming\nexperiences. They started learning Java at school, and it soon turned out to\nbe horrible.\nA lot of us learned to code in languages like BASIC or Pascal. There was no\nobject orientation, no sophisticated file I/O and almost no modularization...\nand it was great. In BASIC you could just write\n\nPRINT "HELLO WORLD"\n\nand you were done. This was actually a running program solving a basic and\nreoccurring problem: Output some text on a screen.\nIf you wanted to do the same thing in Java you just write:\n\npublic class Main {\n public static void main (String[] args) {\n System.out.println("Hello, world!");\n }\n}\n\nDo you see how much knowledge about programming you must have to achieve the\neasiest task one could think of? Describing the program to a novice programmer\nmay sound like this:\n\nCreate a Main class containing a main-method returning void expecting a string\narray as a single argument using the println method of the out object of\nclass PrintStream passing your text as a single argument.\n\nβ€” please just don't forget your brackets. This way your first programming\nhours are guaranteed to be great fun.\nOK. So what are the alternatives? I admit that nobody wants to write BASIC\nanymore because of its lack of a sophisticated standard library for graphics\n(Java doesn't have one either) and its weak scalability. The language has to\nbe clean and straightforward. It should be fast enough for numerical tasks but\nnot as wordy as the rigid C-type bracket languages (sorry C++ guys). It should\nhave a smooth learning curve and provide direct feedback (compiled languages\noften suck at that point). It should encourage clean code and reward best\npractices. One language that provides all that is Python.\nAnd Python has even more: hundreds of libraries that help you with almost\neverything, good integration into common IDEs (PyDev in Eclipse, IDLE...), a\nprecise and elegant syntax.\nHere is our program from above written in Python:\n\nprint("Hello World")\n\nThere's no need to know about object orientation, scopes and function arguments\nat this point. No householding or book-keeping. Yes, it's an interpreted\nlanguage, but that's not a deal breaker for beginners.\nIf you aren't convinced yet, printing and formatting text output in Java is\nrelatively easy for an advanced programmer but the gruesome stuff begins with\nfile input:\n\nimport java.io.BufferedReader;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.IOException;\n\npublic class fileIO {\n public static void main(String[] args) {\n String filename = "test.txt", line;\n try {\n BufferedReader myFile =\n new BufferedReader(new FileReader(filename));\n\n while ( ( line = myFile.readLine()) != null) {\n System.out.println(line);\n }\n } catch (FileNotFoundException e) {\n e.printStackTrace();\n } catch (IOException e) {\n e.printStackTrace();\n }\n }\n}\n\nI hear you say: "Dude, file I/O is pretty complex. It's just the way it is".\nThat's true... internally . But a beginner should get an easy interface.\nPython shows how it's done:\n\nfile = open("test.txt")\ntext = file.read()\nprint(text);\n\nThe code goes hand in hand with the natural understanding of how the process\nworks: "The computer opens a file, reads it and prints it". Even a five-year-old\nkid can understand that. Nobody would start to explain: "Before you can read a\nfile you need a BufferedReader that works on a FileReader..." even if this is\nprecisely how it works internally. You want to explain the big picture at\nfirst. The elementary principles of teaching a computer how to do useful stuff.\nOtherwise, you will start frustrating beginners and fool them into thinking that\nthey are not bright enough for programming. Programming is fun and starting with\nit is the most crucial step. So don't spoil that experience with layers of\nunneeded abstraction.\n\n \n \n \n \nLinks\n\nResponse to this article (almost ten years later): Why Kotlin may be better than Java and Python as the first programming language\n\n" }, { "title": "Howto Sort a Vector or a List in C++ using STL", "url": "https://endler.dev/2010/howto-sort-a-vector-or-a-list-in-c-using-stl/", "body": "A little code snippet that people need very often.\n\n/*\n* Howto sort a vector or a list in C++ using STL\n*/\n\n#include <algorithm> // Needed for sort() method\n#include <vector> // STL vector class\n#include <list> // STL list class\n#include <iostream> // Needed for cout,endl\n\nusing namespace std; // Save us some typing\n\n/*\n* This is a comparison function. It can be used to tell sort()\n* how to order the elements in our container (the vector or list).\n* You can write a comparator for every data type (i.e. double, string...).\n*/\nbool comp(const int& num1, const int& num2) {\n return num1 > num2;\n}\n\nint main() {\n // SORTING WITH VECTORS //\n\n // A vector containing integers\n vector<int> v;\n\n // Insert some values\n v.push_back(5);\n v.push_back(12);\n v.push_back(1);\n\n // The generic STL sort function uses three parameters:\n // \n // v.begin() Iterator pointing at the _beginning_ of the container\n // v.end() Iterator pointing at the _end_ of it\n // comp [Optional] A comparison function (see above)\n // \n // The above mentioned iterators must be random access iterators because\n // sort() takes advantage of clever tricks that require direct access to\n // all elements of the vector. This makes it really fast.\n // (Currently introsort is used with O(n*log n) even in worst case).\n\n sort(v.begin(), v.end(), comp);\n\n cout << "Vector: ";\n\n // Iterate over vector elements\n vector<int>::iterator vIt;\n for (vIt = v.begin(); vIt != v.end(); vIt++) {\n // Print current element to standard output\n cout << *vIt << " ";\n }\n cout << endl;\n\n // SORTING WITH LISTS //\n // A list containing integers\n list<int> l;\n\n // Insert some values\n l.push_back(5);\n l.push_back(12);\n l.push_back(1);\n\n // Here is the major difference between vectors and lists in general:\n // Vectors offer fast random access to every element\n // but inserting a new element at the beginning or in the middle is slow.\n // On the other hand inserting into a list is fast but searching for\n // a specific element is slow.\n //\n // Vectors behave much like an array, while lists only allow slow sequential access.\n // Therefore we need a different function to sort all elements that does\n // not need random access iterators.\n // \n // comp [Optional] A comparison function (see above)\n // \n // Note that sort() is specific for the list and is implemented as a\n // member function of list<>. This is feels more object oriented than the vector.\n \n l.sort(comp);\n\n cout << "List: ";\n\n // A pointer to a list element\n list<int>::iterator lIt;\n for (lIt = l.begin(); lIt != l.end(); lIt++) {\n cout << *lIt << " ";\n }\n cout << endl;\n\n return 0;\n}\n\n \n \n \n \nCompilation and execution\nSave the above code inside a file, e.g. list_vector.cpp and compile it like so:\n\nclang++ list_vector.cpp\n\nTo run it, execute the resulting binary.\n\n./a.out\n\n \n \n \n \nProgram output\n\nVector: 12 5 1\nList: 12 5 1\n" }, { "title": "Why I Love Text Files", "url": "https://endler.dev/2010/why-i-love-text-files/", "body": "Text files are the single most important way we can communicate with computers. It's no coincidence that they are also the most vital way to interact with other human beings. What we can achieve with text files is invaluable: Write it once and refer to it whenever you want to get the message across in the future. Write a program (it's just text), save it and let the machine execute it whenever you like. Write another text file which contains the rules for the execution of your program and the computer runs your application exactly as you specified (cron files do that on Unix).\nText files can be structured in any way you can imagine. Some flavours are JSON, Markdown and SVG. It's all just text. There exist a billion of programs and algorithms to access, modify and distribute text files. You can write them with Emacs, print them on a terminal, pipe them through sed and send them via email to a friend who publishes them on the web. Because text files are so important we have good support for them on any computing system. On Unix, everything is a file and HTML is just structured text. It's a simple and powerful tool to make a contribution to society that outlasts our lives.\nI have a single text file in my mac dock bar which is called TODO.txt. I open it every day, and after years of experimenting with different task management apps from simple command line tools to sophisticated online information storage systems, I always came back to plain text files. And the explanation is simple: If humanity will still be around a thousand years from now, chances are that plain text files are one of the very few file formats that will still be readable.\nThey are an incremental part of how we can modify our environment without even leaving our desk. They have no overhead and can contain a single thought or the complete knowledge of our species. Distributing textual information is so vital for us that we permanently develop faster distribution networks – the fastest by now being the internet.\nOn the web, you have instant access to a virtually endless amount of information and data distributed as plain text files. New web services made accessing the data even easier offering APIs and feeds. You can pull down the data from their servers and make statistics with a programming language of your choice. As you may have noticed, my affinity to text files partially comes from my programming background. As Matt Might correctly points out on his blog:\n\nThe continued dominance of the command line among experts is a testament to the power of linguistic abstraction: when it comes to computing, a word is worth a thousand pictures.\n\nWhenever you like a text on the web, just link to it and create a wonderful chain of ideas. Want to read it later or recommend it to a friend? Just share the text or print it on paper. The fact that we all take such things for granted is a testament for the power of text files and their importance for the information age.\n\n \n \n \n \nLinks\n\nUnix Text Processing (PDF)\n\n" }, { "title": "Running Legacy Code", "url": "https://endler.dev/2009/running-legacy-code/", "body": "This short article deals with a severe problem in software development: bit rot.\nWhen switching to a new platform (for instance from Windows XP to Windows Vista/7), the programmers need to make sure that old bits of code run flawlessly. There are several ways to achieve this goal that will be discussed in the next paragraphs:\n\n \n \n \n \nPorting the code\nThis is generally considered a hard path to follow. For non-trivial legacy code-blocks, chances are high that they contain side-effects and hacks to make them work in different environments. Porting code means replacing parts of the program that use functions and methods that don't exist anymore with new ones which make use of the modern librariesΒ  and routines of the new platform. The significant advantages are maintainable software and sometimes faster running programs. But it may be needed to hack the new platform libraries in order to preserve the whole functionality of an old application. When changing an algorithm inside legacy code, the ported version may become unstable. Thus there may be better ways of maintaining obsolete code today.\n\n \n \n \n \nEmulators\nEmulators work much the same like porting the code. You replace old function calls with new ones to make everything work again. However you don't alter the old codebase itself (because you may not have the source code available) but you create a new compatibility layer that "translates" the communication between the underlying operating system and software (our new platform) and our old software. Emulation can also be very fast and run stable for many years but writing an emulator can be even harder than porting the code because an educational guess may be needed to figure out how the program works internally. Additionally, the emulator itself may become obsolete in the future and might eventuallyΒ  be replaced by a new one.\n\n \n \n \n \nVirtual machines\nDuring the last years, a new approach was gaining popularity. The idea is simple: Don't touch anything. Take the whole platform and copy it in order to run old software. The old software runs on top of the old operating system within a virtual machine that runs on the new platform.\nFrom a sane software developers view, this method is ridiculous. A lot of resources are wasted along the way. The system is busier switching contexts from an old platform to the new one and back than running the actual legacy program. However, with cheap and capable hardware everywhere this idea gets more and more interesting. As Steve Atwood coined it:\n\nAlways try to spend your way out of a performance problem first by throwing faster hardware at it.\n\nAnd he's right. The Microsoft developers did the same on their new NT 6.0 platform (Vista, Windows 7, Windows Server 2008...): Windows XP is running on a virtual machine. This way everything behaves just like one would run the software on the old system. And by optimizing the performance bottlenecks (input/output, context switches), one gets a fast and stable, easy to maintain product.\nEvery method has its major advantages and disadvantages. It's on the developer to select the appropriate strategy.\n" } ]