< prev 12 Feb 2015 to 14 Nov 2014 next >

Posts tagged 'algorithms'

  • Skill Cap

    I've just started to read Corman's Algorithms, and in the first chapter a thought occurred to me.

    There was a recent post about how inter-connected the Facebook codebase is. First of all, I found it quite surprising how much code there is, but I assume a good chunk of it is for scalability, which is certainly important (it's basically a mystery to me how facebook works at all, given that there's no way their user database can exist as a single object on a single machine).

    However, thinking further, and thinking about the sorts of companies I see out there, it seems like there are a lot of people solving the same kinds of problems. And between one thing and another, I started wondering: how much of this is because our languages for expressing common problems haven't matured?

    There's a cap on median engineer intelligence (it doesn't really matter what this means, but for the sake of argument, let's say the median engineer working in web technologies has an IQ no higher than 130, or two sigma above the population). Engineers are in a weird spot inasmuch as they work in one of the few industries where your job is very similar to your training and tool-building. A teacher can take classes on being a better teacher, and make resources to make teaching better and easier, but at the end of the day if they don't get in front of a classroom, they're not doing their job.

    An engineer, on the other hand, can write a framework to make their work easier, learn (or create) a domain-specific language to speed up one aspect of their job that's repetitive, and at the end of the day have in hand new knowledge, work that pays the bills, and a tool they can release for other developers to use. This probably goes a long way in explaining the galapagosation of technologies, especially web technologies.

    At some point, the learning curve of understanding how everything goes together becomes too challenging. Thus, our median engineer can't get much more efficient, because there are too many things to keep track of, and too many things that can break, and he wants to understand his toolchain and not look a fool because he can't tell his boss why the database got borked.

    The thing is, we already have specialists (Database Engineers, SysAdmins, Front-end, Back-end, Project Engineers, the list goes on), and the role of "DevOps" kind of papers over the fact that at some scale, a company should be concerned with differentiation of responsibilities. All engineers should have some idea of how their work might affect other domains, but the job of optimization should lie in the hands of experts. (Note here Conway's Law, which states that "organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations".)

    Is it possible to make a framework which would make engineers more productive, and capable of expressing higher-order concepts without having to worry about the entire stack?

    One attempt at an answer: the sorts of problems that companies are focused on solving largely center around CRUD, certainly. We're getting to a point where most of the things people want to accomplish involve finding a collection of rows in a database, and presenting them in a way that makes sense to users. Visual pattern matching, just-in-time data delivery, etc, are all very important to making a successful application. But when all the code you use on the front-end is visible to users (via JavaScript) and the actions people take are generally similar across the board (CRUD), creating a niche relies on largely on branding and network effects.

    The original "killer app" on Facebook was (probably) checking out the pictures of cute girls in your classes, and bragging about your awesomeness in an attempt to improve social standing, and not much has changed. They leverage more aspects of homo now, ten years on—the desire to be informed, the desire to feel social without the costs—but these things depend on the network that was bootstrapped via "cool, attractive young people".

    And yes, I'm being a bit glib, but the problems that Facebook solves in interesting ways mostly center around speed, availability, scale, and presentation. The CRUD core is kind of… boring?

    The company that eats Amazon's lunch, by contrast, will make it easier to actually shop, and not just find something close. This is presentation and filtering layer technology, and Amazon's existence hasn't shut the door on other online marketplaces the way Facebook has largely supplanted all other entrants to vanilla social networking.

    Amazon's power has come, in turn, from its infrastructure and its internal APIs. Most engineers, when they hear "Amazon" now, will think more naturally of AWS than of the original online bookseller. Sure, Amazon will always mean books on some level, but they drive so. much. of the web that I wouldn't be surprised if that becomes their core business (just like I joke about Google becoming primarily a car manufacturer).

    [A funny sidenote: how many online stores, I wonder, are running on AWS on the backend? Amazon is pretty content-agnostic, even when the instances they're hosting would compete with their original niche.]

    Meanwhile, technology stacks are maturing—redis for cache, capistrano for server management, docker for deployment—thus freeing engineers from having to roll their own, so to speak. Interesting companies are getting more meta (sidenote: I really need to play with SquareSpace to see what they enable people to do… but it's possible that they'll kill the indie web dev)… and I don't actually know what the engineers are doing at the rest of these companies? Is it all just gluing frameworks together, whiling away time until the company is aquihired?

    And there's something empowering to playing the "Full Stack Engineer" game as long as you can. Outside Project Management, FSEs are among the few who can say they're able to realize a vision, which the parts-and-glue, division-of-labor engineers probably can't.

    Other than inertia, then, I suspect a primary reason this industry hasn't settled on a single stack is because as long as there's capital flowing, people are having fun at work—and occasionally making money doing so. If software engineers collectively decided they wanted to change the world in a more meaningful way, they certainly could (or at least could make a good stab at it). However, at that point, the majority of the industry would be consigning themselves to grunt work, the sort of work that mostly appeals to milquetoast personalities with pocket protectors, and late-career people with families and hobbies.

    A business idea, then, would be "do whatever would make software engineering boring, the fastest, for the most people". That would pull the rug out from under the entire field for a while, though, until the middle-road engineers could figure out a way to build on top of this new stack and have fun doing so.

    3:19 PM - 12 Feb 2015 in technology
  • Week 1 Day 5 - Painted into a corner

    Today's partner: Connie

    It's a bit of a challenge to reconstruct my memory of Friday after I failed to save my draft Friday evening, so excuse this entry for being perhaps a bit vague. Also, I'm going to speak in the tense I would have had I completed this entry Friday.

    Today was the first day where there wasn't enough work to actually fill out the day. We had two main projects: a small class that built trees of moves for a Knight navigating a chessboard in order to find the shortest possible route between two points via knight's moves, and a Tic-Tac-Toe solving AI (on top of a couple small vignettes). Of course, the point of the AI wasn't to show any great details about AIs; rather, we were working more on Object Oriented programming, data structures, and algorithms.

    These algorithms can broadly be called "paint bucket", or "flood fill" algorithms: starting at a point, do some action to all the "neighbors" (keep in mind that a knight's neighbors on a chessboard are moves of 1,2 away, and not adjacent squares - one of those mathematical things where you have to keep in mind the context you're dealing with), and then repeat for the neighbor's neighbors, ad nauseam until there are no more valid spaces to apply the action.

    Doing this sort of naive AI over and over has certainly helped drive home how to best write recursive OO code to do breadth-first search in different contexts, with different types of objects, but again we're not making contest-winning AIs here.

    Our biggest problem today was a weird edge case bug in the TTT AI. The spec was a bit hard to read, and once we nuked the half-dozen syntax errors in our collected source files, we found the logic as stated in English in the instructions to be a bit vague on what the various parameters means and how to pass them around. It took the better part of an hour or more to vary the code and make the lynchpin methods work properly, but once we got those right, the tests all passed without a hitch, and it was glorious.

    Of course, at that point, we only had an hour left to work. Without a bonus problem to chew on, there wasn't much to be done except to dip our toes into this weekend's reading and talk about the course as a whole. I showed my partner a couple resources I had seen on using git, which I hope she found valuable, and then just chatted about stuff. I know I'll pay for it this weekend in terms of worktime lost, but sometimes you just need a break, you know?

    9:51 PM - 16 Nov 2014 in AppAcademy
  • Week 1 Day 4 - Applied telepathy

    Today's partner: Michael C. (since there are two)

    I think I adapt to things within three days. Today just felt normal, like what I should be doing with my life. I know it's going to get much harder before we finish, and I keep pushing things onto the stack to go into more depth on over the weekend—but there is already plenty to keep me busy. Life's going to be tough enough on its own, and I should probably be getting up early Saturday morning and diving right in. I may have to work remotely to get away from distractions (although this hasn't been as big a problem as I might have thought - Facebook and reddit just aren't very interesting to me now that I have a stack of things to think about).

    I am still not getting enough sleep, and it doesn't seem to actually affect my engagement or 'awareness' during the day. It's an awesome feeling.

    The highlights of the day were working with procs and, again, the big projects toward the end of the day. We were wrestling with understanding how to write our own versions of a couple Enumerable class methods (my_each, my_select, etc) but the breakthrough on this matter came when we tried to express the same things in "normal" ruby code and then translated that work into our functions.

    For instance, normally, to double all elements in an array, you might write

    1

    array.map { |i| i * 2 }

    When defining your own function to receive that proc, you can start with something done in the standard ruby function calls:

    1
2
3
4
5
6
7
8
9

    def my_map(&prc)
  result = []

  self.each do |item|
    result << prc.call(item)
  end

  result
end

    Simply replace "each" with "my_each" and you've basically re-created standard ruby functionality. I assume that, even if it's not now written this way, at some point the ruby "standard library" looked something like this (albeit with more error handling).

    My partner mentioned at the beginning of the day that he didn't really understand why you'd use procs, and within a half hour we both got it.

    The work on recursion was solid practice. The first example took ten minutes, the second five, and each one after was simply an exercise in analyzing the higher-level problem. Satisfying, and excellent practice/pedagogy, but nothing earth-shattering. :)

    make_change, the recursive method for taking a list of coin values (for instance, in the US, 25, 10, 5, and 1 for quarters, dimes, nickels, and pennies) as well as an amount that you're trying to target and yielding the list of coins you'd return in order to make the target amount with the minimum number of coins, was easy, then absolutely maddening. We managed to describe the core logic within a half hour - about one line every 90 seconds, go figure - and then took another half hour in order to track down a crazy edge-case bug that was killing most of our results.

    Constance looked at our code for a good ten minutes before we nailed down what was actually going wrong. We had a specific test case that broke the method quickly, a bunch of ghetto debug statements (p for the win!) and basically total awareness about the state of the internal variables, and we still struggled to find what was causing this failure.

    It turned out that there was a loop condition that this edge case was falling through, returning nil when our calling function was expecting an array. It was such a specific edge case that I should have seen earlier, but I kept looking in the wrong places for this, and the doubt caused by that sort of weakened my learning. I spent five minutes doubting each level of my knowledge, until finally I was doubting whether < actually means "less than". I'll definitely need more practice overall, and to develop more facility with debugger… and I've gained some specific respect for the challenges software maintainers face every day.

    WordChainer, today's "capstone", was a blast. The spec was thorough, and my partner and I basically spent fifteen minutes reading sections of it to one another and getting code into the editor. What a delight a properly detailed spec is! Too bad I'll probably never see one again in my life, unless I write it myself! ;)

    It took us about two hours to get a very respectable and pretty clean class for this portion, overshooting the end of class by a half hour. Remarkably, by the last hour, I wasn't sure which one of us was actually driving or navigating. Michael and I were basically on a single wavelength, and passing the keyboard back and forth was mostly a formality.

    6:02 AM - 14 Nov 2014 in AppAcademy
< prev 12 Feb 2015 to 14 Nov 2014 next >