Posts tagged 'engineering'
-
So. I have a local version of my rails re-write of this middleman blog working. For various reasons I decided that going for full control made the most sense1, and thus I ended up with something that's mostly okay. It's clear that certain doors are open now that weren't before, but it's taken a lot of effort to get here.
One of the habits I wanted to train myself in is in learning to accept external code more readily. I think I'm mostly won on that front, but it's still not perfect. The time it takes to learn an API is not always trivial, and it's not clear when my own assumptions differ from those of the authors' whether I'm correct, or they are. Case in point: I wanted some way to slugify categories and tags, and so went looking for ruby libraries that enable that, and found
friendly_id. It's pretty capable, but one thing I wanted to do wasfirst_or_create_by_slug. After looking through the docs, then the code, for a solution that would make this possible, I started to open a new issue asking for a way to do this. I quickly realized that my request would be meaningless, and I was better off putting together a hack that addressed my own needs, rather than asking for a general purpose solution that wouldn't be a positive addition to the gem.Would I have realized this if I had elected to write my own code, or would I have made some hack that got me halfway only for me to realize that it's completely infeasible?
I am certain that using someone's CSS library (in this case, Zurb's Foundation) was a step forward. I hate worrying about browser compatibility, and at the end of the day I simply want a vocabulary that allows me to express visual ideas in a flexible way. CSS is a nightmare to figure out from source; thus, front-end frameworks offer advantages through documentation, examples, and actually being tested by people who care.
Back to the main thread: I've sort of paused out. This is being written via a web page on my local computer, and… I added two-way data pathways (import/export) from rails so that I can import test data (== my old blog posts) and compose new posts that can then be slurped up by middleman. It's not ideal—it's not the hosted solution I alluded to in an earlier post—but it's quite similar to the "admin app for composing static pages" half-solution I sketched out. The interface, and codebase, are both things I feel comfortable working on, and some of the ancillary effects I was hoping to see have panned out.
That is to say, I'm seeing how I can better (solo) handle problems I haven't seen before (like, finding and using Foundation in the first place), and getting better at judging what is and is not worth my time, even when I don't always close the loop on this knowledge. And adding weird features for personal use—like a button to output a post to markdown—doesn't feel as off-putting to me now as it might have in the past.
On the other hand, I did spend too long getting this site's CSS re-written so I could get the form classes for free, so it might be that the biggest gain I've gotten is clarity. I can see clearly now in all the ways I'm doing things that aren't in my best interests.
But it's all so pretttttty, and flexible!
The wall, now, is that getting the proof-of-concept code/configuration for external hosting and e.g. blue/green is … not a priority anymore? I can output whenever I want, and I've found stopgap ways to track the things I want to say, and while they're not all going up on the web yet, I can see a path that will make that happen in due time.
-
Namely, I want to extend the blog model in ways that aren't easily enabled by most blog platforms/static site generators, and the convolutions I could see happening in the near future were getting extreme. ↩
-
-
- Easy deployment
- Backups, in transferable format, of post content
- Total ownership of content
- Post by email
- Post by web
- Formatting for short posts, versus long form writing
- Easy configuration/additions without
scp(e.g. git push/commit hooks) - Low server overhead—it's a damned static site 99% of the time; it shouldn't require an m4.large
- Article permalinks regardless of underlying categorization/content changes
- Presentation of heterogeneous data
- A way to share more than just text (e.g. images)
- Maintain pagination without incurring excessive backend load
- Portfolio elements—running/presenting arbitrary code, separate from blog content/formatting (e.g. cssris)
- Extensibility
- Easy import/export
-
I've achieved the goal of having something I control, but there's a weakness here: I haven't found an easy workflow that lets me get short thoughts out quickly and painlessly. Yesterday I wanted to write about a half dozen topics that didn't merit a full journal entry, but didn't have an easy way to do it. I ended up just opening a new note on my phone and brain dumping, which isn't the worst thing ever.
I'm trying to avoid using a database, but a lot of the stuff I want to do is most easily solved with access to a database. A forum of this scale doesn't merit a backend, or at least strongly benefits from being statically generated. On my feature list, however, are a lot of things that become easier if I'm running a real server, or at least building the site off something that has a database available.
… Hmm, now that I think of it, that's not a bad solution. The ideal admin tooling suggests the need for a first class backend, but I can easily use that to generate a static site that gets the benefits of both worlds.
Spitballing here, there are a couple decent ways to get a simple, high speed personal website running in the hosting environment I have available.
- Run a tiny app for one user that has reasonable login protections. When I hit a button, dump some form of the content into a format that a static site generator understands, and have that generator deploy the content to my preferred host (could be S3, could be similar to the current configuration).
- Wire up a tiny blog app that caches aggressively when not signed in. I've seen some of this magic in a previous life; the nginx config used would serve assets directly from nginx instead of routing through rails, boosting app speed. If there's a clean way to have rails do that automatically, I might go that route.
- Use normal app-level caching and hope for the best. I don't much like this one because I would have to do a lot of testing ahead of time to ensure that I could serve hundreds or thousands of requests for the same page without hitting server bottlenecks.
Taking a step in any of these directions will change the calculus I've been using so far. Notably, it's been easy (or, relatively easy) to bulk import/edit preexisting posts to work with this tech stack, but having a database at a distance will make it harder. Since I should be doing more original writing and less migrations as time goes on, the balance of burden will shift.
Looking back at my decisions up to this point, the following facts are true:
- I'm happy that I've kept data, configuration, and code separate so far; it will make any migration to a different platform easy. Many existing frameworks for producing static sites encourage too much commingling of these things.
- The mere existence of this blog has been a great boon, in the sense that I'm writing at all.
- The current form of this site is essentially an MVP: I've got posts on a screen that can be viewed on the internet. In that sense, it's a win so far. It looks like any path forward will demand more backend work and server orchestration, which was half the purpose of this exercise in the first place. I can't say, then, that I'm upset with this state of affairs.
I can't overstate how much having this bootstrapped site up and running has been for my mental well being. I feel lately like I'm overflowing with ideas.
-
I've just started to read Corman's Algorithms, and in the first chapter a thought occurred to me.
There was a recent post about how inter-connected the Facebook codebase is. First of all, I found it quite surprising how much code there is, but I assume a good chunk of it is for scalability, which is certainly important (it's basically a mystery to me how facebook works at all, given that there's no way their user database can exist as a single object on a single machine).
However, thinking further, and thinking about the sorts of companies I see out there, it seems like there are a lot of people solving the same kinds of problems. And between one thing and another, I started wondering: how much of this is because our languages for expressing common problems haven't matured?
There's a cap on median engineer intelligence (it doesn't really matter what this means, but for the sake of argument, let's say the median engineer working in web technologies has an IQ no higher than 130, or two sigma above the population). Engineers are in a weird spot inasmuch as they work in one of the few industries where your job is very similar to your training and tool-building. A teacher can take classes on being a better teacher, and make resources to make teaching better and easier, but at the end of the day if they don't get in front of a classroom, they're not doing their job.
An engineer, on the other hand, can write a framework to make their work easier, learn (or create) a domain-specific language to speed up one aspect of their job that's repetitive, and at the end of the day have in hand new knowledge, work that pays the bills, and a tool they can release for other developers to use. This probably goes a long way in explaining the galapagosation of technologies, especially web technologies.
At some point, the learning curve of understanding how everything goes together becomes too challenging. Thus, our median engineer can't get much more efficient, because there are too many things to keep track of, and too many things that can break, and he wants to understand his toolchain and not look a fool because he can't tell his boss why the database got borked.
The thing is, we already have specialists (Database Engineers, SysAdmins, Front-end, Back-end, Project Engineers, the list goes on), and the role of "DevOps" kind of papers over the fact that at some scale, a company should be concerned with differentiation of responsibilities. All engineers should have some idea of how their work might affect other domains, but the job of optimization should lie in the hands of experts. (Note here Conway's Law, which states that "organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations".)
Is it possible to make a framework which would make engineers more productive, and capable of expressing higher-order concepts without having to worry about the entire stack?
One attempt at an answer: the sorts of problems that companies are focused on solving largely center around CRUD, certainly. We're getting to a point where most of the things people want to accomplish involve finding a collection of rows in a database, and presenting them in a way that makes sense to users. Visual pattern matching, just-in-time data delivery, etc, are all very important to making a successful application. But when all the code you use on the front-end is visible to users (via JavaScript) and the actions people take are generally similar across the board (CRUD), creating a niche relies on largely on branding and network effects.
The original "killer app" on Facebook was (probably) checking out the pictures of cute girls in your classes, and bragging about your awesomeness in an attempt to improve social standing, and not much has changed. They leverage more aspects of homo now, ten years on—the desire to be informed, the desire to feel social without the costs—but these things depend on the network that was bootstrapped via "cool, attractive young people".
And yes, I'm being a bit glib, but the problems that Facebook solves in interesting ways mostly center around speed, availability, scale, and presentation. The CRUD core is kind of… boring?
The company that eats Amazon's lunch, by contrast, will make it easier to actually shop, and not just find something close. This is presentation and filtering layer technology, and Amazon's existence hasn't shut the door on other online marketplaces the way Facebook has largely supplanted all other entrants to vanilla social networking.
Amazon's power has come, in turn, from its infrastructure and its internal APIs. Most engineers, when they hear "Amazon" now, will think more naturally of AWS than of the original online bookseller. Sure, Amazon will always mean books on some level, but they drive so. much. of the web that I wouldn't be surprised if that becomes their core business (just like I joke about Google becoming primarily a car manufacturer).
[A funny sidenote: how many online stores, I wonder, are running on AWS on the backend? Amazon is pretty content-agnostic, even when the instances they're hosting would compete with their original niche.]
Meanwhile, technology stacks are maturing—
redisfor cache,capistranofor server management,dockerfor deployment—thus freeing engineers from having to roll their own, so to speak. Interesting companies are getting more meta (sidenote: I really need to play with SquareSpace to see what they enable people to do… but it's possible that they'll kill the indie web dev)… and I don't actually know what the engineers are doing at the rest of these companies? Is it all just gluing frameworks together, whiling away time until the company is aquihired?And there's something empowering to playing the "Full Stack Engineer" game as long as you can. Outside Project Management, FSEs are among the few who can say they're able to realize a vision, which the parts-and-glue, division-of-labor engineers probably can't.
Other than inertia, then, I suspect a primary reason this industry hasn't settled on a single stack is because as long as there's capital flowing, people are having fun at work—and occasionally making money doing so. If software engineers collectively decided they wanted to change the world in a more meaningful way, they certainly could (or at least could make a good stab at it). However, at that point, the majority of the industry would be consigning themselves to grunt work, the sort of work that mostly appeals to milquetoast personalities with pocket protectors, and late-career people with families and hobbies.
A business idea, then, would be "do whatever would make software engineering boring, the fastest, for the most people". That would pull the rug out from under the entire field for a while, though, until the middle-road engineers could figure out a way to build on top of this new stack and have fun doing so.