Philip ReynoldsRamblingsJekyll2023-01-11T15:32:46+00:00https://brassy.net/Philip Reynoldshttps://brassy.net/philip.reynolds@gmail.com"https://brassy.net/articles/the-job2023-01-10T00:00:00+00:002023-01-10T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<p>As I’ve taken up different fitness routines, sports, diets etc. throughout the
years, I’ve gotten experience with what the hard thing about them all are. And
it’s not picking them. It’s doing them. </p>
<p>The most likely reason for me not to be fit or healthy wasn’t ever that I picked
the wrong strength training programme. It was that I had either procrastinated
so long on the right one to do that I never did anything, or I had fallen off
the wagon and stopped the weekly schedule I had laid out. </p>
<p>With a little more grey hair in my professional career, I’ve started to call
this “the job”. </p>
<p>The job is not to figure out the perfect fitness plan. Find one
quickly that looks good enough. The actual job is to set myself up for success
against executing against that regularly.</p>
<h1 id="nudges">Nudges</h1>
<p><a href="https://en.wikipedia.org/wiki/Nudge_(book)">Malcolm Gladwell’s Nudge</a> describes
some techniques for helping push us in the right direction. Set up your
environment & routine for success. Leave your gym clothes at the end of the bed.</p>
<p>These sound so simple that you could almost overlook them. I’ve found them to be
the most critical tools in driving outcomes. Everyone wants to be “strategic”.
Most people undervalue execution. </p>
<ul>
<li>
<p><a href="https://blog.noisli.com/what-it-means-to-eat-the-frog/">Eat the frog</a>. Do the most difficult thing first when your motivation is at it’s highest. This doesn’t work with all tasks where you can’t reorder. </p>
</li>
<li>
<p>Write down a plan. Review frequently. Calendar in the review so that it gets
done. It was amazing to me early in my career how effective create a small
list of things that need to be done and ticking them off one by one was to my
motivation. </p>
</li>
<li>
<p>Take the smallest step on the journey. eg Calendar the meeting.
Procrastinating on a personnel issue? Send HR an email.</p>
</li>
<li>
<p>Share a plan with someone who will hold you accountible. Maybe your boss or a
mentor and commit to reviewing it.</p>
</li>
</ul>
<h1 id="figuring-out-the-job">Figuring out the job</h1>
<p>Sometimes, the hardest thing is figuring out “the job”.</p>
<p>The job in software development is delivery. Building things on laptops is the
easy bit. Getting and running things in production is hard.</p>
<p>The job when you inherit an oganisation is generally about making the personnel
changes you need to make. Swiftly. Most people overindex on the plan and don’t
emphasize enough how are they and their organisations going to execute. Most
plans I’ve seen identify most of the personnel issues correctly. If you come
back to them in a year though, it’s amazing how many have not been actioned.</p>
<p>The job is frequently the unsexy, unloved part of what needs to be done. It
often does not look like the important part from the outside. It’s rarely, if
ever, strategic. It frequently is more doing than thinking and It frequently
requires discipline, consistency or other boring words that require something to
be done again and again over time.</p>
<p>If you’re still unsure then the most likely source of information here is
experienced practitioners.</p>
<p>“Amateurs talk strategy. Professionals talk logistics”</p>
<p><a href="https://brassy.net/articles/the-job">The Job</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on January 10, 2023.</p>https://brassy.net/articles/aws-inventory2022-11-03T00:00:00+00:002022-11-03T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<h2 id="inventory">Inventory</h2>
<p>One of the problems when you inherit something is not knowing fully what’s
where.</p>
<p>While I’m sure there are probably lots of other reasons for wanting a full
comprehensive inventory from AWS </p>
<p><a href="https://brassy.net/articles/aws-inventory">Trying to get AWS inventory</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on November 03, 2022.</p>https://brassy.net/articles/due-diligence-checklist-b2b-saas2022-11-01T00:00:00+00:002022-11-01T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<h2 id="the-list">The List</h2>
<ul>
<li><strong>1-00 Legal</strong>
<ul>
<li><strong>1-01</strong> - Trademark Information</li>
<li><strong>1-02</strong> - Entity details who owns current assets</li>
<li><strong>1-03</strong> - Domain Name(s)</li>
<li><strong>1-04</strong> - Other Source Identifiers (Social media handles)</li>
<li><strong>1-05</strong> - Inventory of software & databases</li>
<li><strong>1-06</strong> - Copyrights</li>
<li><strong>1-07</strong> - Patents</li>
<li><strong>1-08</strong> - Details of prior sale (if applicable)</li>
<li><strong>1-09</strong> - Claims / Litigations / Complaints</li>
</ul>
</li>
<li><strong>2-0 Finances</strong>
<ul>
<li><strong>2-01</strong> - P&L (36 months)</li>
<li><strong>2-02</strong> - Income Statement (36 months)</li>
<li><strong>2-03</strong> - Stripe Account or equivalent (read-only account)</li>
</ul>
</li>
<li><strong>3-00 Privacy & Security</strong>
<ul>
<li><strong>3-01</strong> - Privacy Policies</li>
<li><strong>3-02</strong> - PII & User Data Collection</li>
<li><strong>3-03</strong> - Security Breaches</li>
<li><strong>3-04</strong> - Software inventory</li>
<li><strong>3-05</strong> - System related information</li>
<li><strong>3-06</strong> - AWS inventory</li>
<li><strong>3-07</strong> - GDPR / CCPA Compliance</li>
<li><strong>3-08</strong> - ToS</li>
<li><strong>3-09</strong> - Credential / Secret information</li>
</ul>
</li>
<li><strong>4-00 Tech</strong>
<ul>
<li><strong>4-01</strong> - Read-only access to the source code</li>
<li><strong>4-02</strong> - Inventory of all services used</li>
<li><strong>4-03</strong> - Third party services</li>
<li><strong>4-04</strong> - Third party software</li>
<li><strong>4-05</strong> - Software Architecture documentation</li>
<li><strong>4-06</strong> - Deployment information</li>
</ul>
</li>
<li><strong>5-00</strong> - Customers, Sales & Marketing
<ul>
<li><strong>5-01</strong> Market information including any market research</li>
<li><strong>5-02</strong> Information related to product usage (read-only access to
analytics accounts / dashboards etc.)</li>
<li><strong>5-03</strong> Customer Support Volume</li>
<li><strong>5-04</strong> Customer Support Content (recent history)</li>
<li><strong>5-05</strong> </li>
</ul>
</li>
<li><strong>6-00</strong> - Vendor information
<ul>
<li><strong>6-01</strong> - Full vendor contact details (key suppliers, </li>
<li><strong>6-02</strong> - Contractor usage information including previous bills and/or
timesheets</li>
</ul>
</li>
</ul>
<p>For small asset purchases, it may be that there isn’t an independent P&L or
Income statement that can be pulled out for the entity. The inventory section
becomes critical here to make sure that all expenses are accounted for.</p>
<h2 id="some-process">Some process</h2>
<p>I created two documents:</p>
<ol>
<li>A spreadsheet like the one below here which tracked each line item and the
status on both the buyer and seller side.</li>
<li>An accompanying Google doc which detailed each of the sections above and
allowed the seller to input information where relevant so we could share.</li>
</ol>
<p>These documents were shared between us in the data room.</p>
<p><img src="/images/checklist-screenshot.jpg" alt="Checklist spreadsheet" /></p>
<p><a href="https://brassy.net/articles/due-diligence-checklist-b2b-saas">Due Diligence Checklist for B2B SaaS</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on November 01, 2022.</p>https://brassy.net/articles/management-decision-making2016-01-11T00:00:00+00:002016-01-11T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<h2 id="decisions-decisions">Decisions, decisions</h2>
<p>One of the things that struck me most about moving in to management
originally was just how many decisions you were involved in. Whether it was
decisions you were personally making for yourself, or your team, team
decisions that you are helping to make or decisions your team have an opinion
on but you ultimately have a casting vote on.</p>
<p>In a busy environment, you often don’t have the required time to simply sit
and think ad nauseum about every decision you need to make. Retrospectively
looking at every decision is also typically a luxury.</p>
<p>Moving quickly on decisions, where possible, is absolutely necessary. That
means using established patterns to enable yourself and your team to make
decisions more confidently.</p>
<h2 id="reversibility">Reversibility</h2>
<p>There will be a number of times where you find yourself or your team in a
decision making quandry. One of the first questions I try to ask is how easily
can we reverse this? If the answer is quickly, then we try to move past that
quickly. Don’t be afraid to introduce this concept to your teams either. By
simply introducing that concept in to a discussion, dogma often quickly
disappears. </p>
<p>Decisions that aren’t reversible - the metaphorical <a href="https://en.wikipedia.org/wiki/Judgment_of_Solomon#.22Splitting_the_baby.22">‘cutting the baby in
half’</a>
- and have high impact may need to be given more consideration. </p>
<p>Hiring? Not easily reversible.
Architecture choice? Not easily reversible.
Workflow? Easily reversible.
Button colour? Easily reversible.</p>
<p>The next time you end up in a discussion that seems like it’s going nowhere,
ask yourself and your team can you easily reverse a decision and if so, try
and move forward quickly.</p>
<h2 id="introducing-reversibility">Introducing reversibility</h2>
<p>There are times where you can introduce the property of reversibility to a
decision making process. </p>
<p>A technical example of this is using a
<a href="https://en.wikipedia.org/wiki/Facade_pattern">Facade</a> on a third party library.
Leaky abstractions aside, the ability to reverse that decision later by
replacing the underlying concrete implementation can be beneficial. </p>
<p>This doesn’t eliminate the effort in reversing that decision, but it may
significantly reduce it.</p>
<p>A vendor example, might be hiring a consultant or firm to do a once off
smaller job, before making a decision to enter in to a longer term
higher-value agreement.</p>
<p>Don’t just hope for reversibility to be present. You can often introduce it.</p>
<h2 id="reversibility-and-agile">Reversibility and Agile</h2>
<p>I should note that this idea of reversibility is very present in all <a href="http://agilemanifesto.org/">agile
methodologies</a>. Deliver in small increments,
allowing us to change course once we receive feedback on a small, delivered
piece of product or functionality. </p>
<p>It’s not always possible, but it’s one tool to put in your decision making
toolbelt! </p>
<p><a href="https://brassy.net/articles/management-decision-making">Management, decision making and cutting the baby in half</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on January 11, 2016.</p>https://brassy.net/articles/productize-your-infrastructure2015-12-12T00:00:00+00:002015-12-12T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<p>The sudden uptick & interest in “devops” over the past few years is driven by
Cloud. With a capital C. Historically, operations built out the infrastructure
and then slapped the application on top of it that the developers had
produced. As your application gets bigger and so does your infrastructure,
things get harder. Welcome to the Cloud Forcing Function.</p>
<p>In a world where people talk about Infrastructure-as-a-Service, the real key
to growing at <em>scale</em> is <em>Productizing your Infrastructure</em>. </p>
<h2 id="what-do-we-mean-by-productizing">What do we mean by Productizing?</h2>
<p>Let’s start with the definition of a product. Put simply, a product is
something that is manufactured for sale or distribution. However, when we talk
about building software, there are a lot more implications.</p>
<ul>
<li>It’s designed</li>
<li>It can be repeatably built and delivered</li>
<li>It has tangible features</li>
<li>Each component is tested and has some level of quality</li>
</ul>
<p>Put another way, the way we build software systems should also be the way we
build the Infrastructure. Think
<a href="https://en.wikipedia.org/wiki/Systems_development_life_cycle">Systems Development Life Cycle</a></p>
<p>Building your infrastructure should be approached no differently to the way we build
software. </p>
<p>Quite simply, get out of the business of <em>doing stuff</em> and into the business
of <em>building stuff</em>. </p>
<h2 id="symptoms-of-not-having-products">Symptoms of not having products</h2>
<ul>
<li>Repeated requests for the same item over and over </li>
<li>Making direct changes to live running servers & network devices </li>
<li>Frequently having to revisit issues that you thought you had resolved</li>
</ul>
<p>Paul Graham, an extremely prominent writer & startup accelerator founder,
talks specifically <a href="http://www.paulgraham.com/startupfunding.html">about the idea of startups being high growth focussed on
scale and says…</a></p>
<blockquote>
<p>To be a startup, a company has to be a product business, not a service
business. By which I mean not that it has to make something physical, but that
it has to have one thing it sells to many people, rather than doing custom
work for individual clients. Custom work doesn’t scale. </p>
</blockquote>
<h2 id="productize-your-consulting-work">Productize your consulting work</h2>
<p>Simply building automation around common tasks is not the same as building a
product. Automation is internal to you. You need to have something that you
hand to your customers and step back from.</p>
<ul>
<li>Start managing your <a href="http://en.wikipedia.org/wiki/Work_in_process">Work in Progress</a></li>
<li>Start identifying work that is frequent and repititve </li>
<li>Distill one piece of work down to the <a href="https://en.wikipedia.org/wiki/Minimum_viable_product">smallest possible piece that can deliver
value</a></li>
<li>Build software to deliver that value to a customer</li>
<li>Wash, Rinse, Repeat…</li>
</ul>
<p><a href="https://brassy.net/articles/productize-your-infrastructure">Productize your Infrastructure</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on December 12, 2015.</p>https://brassy.net/articles/getting-in-to-devops2015-11-24T00:00:00+00:002015-11-24T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<h2 id="danger">Danger!</h2>
<p>My buzzer just went off when you used the word ‘Devops’. Ewww! Devops is not
a moniker I’m fond of for many reasons. However, that’s not your fault. Given
how prevalent the term is, let’s ignore that for the time being.</p>
<p>One of the main challenges with this denomination is the variety of different
things people mean when they use it. Mostly somewhere between a build engineer
& web operations engineer. My definition of what folks are looking for are
Linux System Engineers who can code to a high level. I personally wanted to be
an engineer who can build maintainable, scalale systems by either writing
code, using 3rd party products or, more often than not, some mixture of the two.</p>
<p>No matter what your definition is, whenever you want to be something in the
tech world, I have some generic advice. Make it your passion. Build your
craft. I gave a generic “career talk” for <a href="http://www.careerzoo.ie/">CareerZoo</a>
previously which talks about <a href="http://www.slideshare.net/PhilipReynolds4/your-career-a-hiring-managers-perspective">this idea of passion for your
career</a>.</p>
<p>In doing a bit of research for this blog post, I came across <a href="https://www.youtube.com/watch?v=LAP1zaXUvAE">this talk from
Theo Schlossnagle</a>. The idea of
having passion for what you do and becoming <em>obsessed</em> with the craft is very
very key. Understand that success in an industry with so much to learn is
difficult and is becoming more difficult by the years with more technology.</p>
<h2 id="forewarning">Forewarning</h2>
<p>Some of these bullet points are weeks worth of effort. Some are months.
Some are years. To really master what I’m talking about, it’s a slow journey.
The nice thing is it can be very practical and I think that’s what is very
attractive to people who get in to stuff via this route.</p>
<p>The only thing I would recommand is starting the programming side as soon as
possible.</p>
<p>This is not meant to be exhaustive or comprehensive. You will have to join
plenty of dots yourself and you’ll obviously pick up lots of other things
along the way. The key is to never stop learning and try to apply your
knowledge as quickly as you learn it. </p>
<p>Pick a project or two at the bottom and try it out. </p>
<h2 id="systems-engineering">Systems Engineering</h2>
<ul>
<li>Start learning how to install and administer Linux boxes
<ul>
<li>Pick your favourite distribution. If you don’t have one, pick Ubuntu.</li>
</ul>
</li>
<li>Setup your own web servers, dns servers, email servers and run them on
the internet.
<ul>
<li>Not sure which ones? Apache, Bind & Postfix.</li>
</ul>
</li>
<li>Start learning how to use the shell like an expert.</li>
<li>Pick up an editor - vim’s fairly ubiquitous. Emacs is another alternative.
<ul>
<li>If you don’t care, pick vim.</li>
</ul>
</li>
<li>Understanding networking
<ul>
<li>Know how IP’s, gateways, netmasks all work. </li>
<li>Understand CIDR.</li>
<li>Understand ARP, ICMP, traceroute</li>
</ul>
</li>
<li>Learn a monitoring system.
<ul>
<li>Understand how to monitor your environment.</li>
</ul>
</li>
<li>Learn package management.
<ul>
<li>Understand how your distribution of choice packages up software. </li>
<li>Understand </li>
</ul>
</li>
<li>Learn an RDBMS
<ul>
<li>If you’re not sure, pick PostgreSQL.</li>
</ul>
</li>
<li>Understand filesystems
<ul>
<li>Start with simple local filesystems (ext3, ext4)</li>
<li>Learn NFS (as much as that is possible)</li>
<li>Look at distributed filesystems like Ceph, Gluster </li>
</ul>
</li>
</ul>
<h2 id="configuration-management">Configuration Management</h2>
<ul>
<li>Learn a config management system
<ul>
<li>If you don’t care, use ansible</li>
</ul>
</li>
<li>Practice trying to deploy stuff with ansible
<ul>
<li>Use 3rd party stuff to start, but understand and learn how to write your
own playbooks.</li>
</ul>
</li>
</ul>
<h2 id="learning-to-code">Learning to code</h2>
<ul>
<li>Learn how to write bash comfortably
<ul>
<li>You should be able to parse most log files fairly easily.</li>
<li>awk & sed should be second nature to you</li>
<li>Automate random duties (e.g. backups) using a config mgmt system</li>
</ul>
</li>
<li><a href="http://learnpythonthehardway.org/">Learn python</a>
<ul>
<li>You need to be a polyglot & know multiple languages. </li>
<li>Powerful, dynamic scripting language. </li>
<li>Often great for when bash isn’t.</li>
<li><em>Master the language</em></li>
</ul>
</li>
<li>Learn C
<ul>
<li>You don’t necessarily need to be a C master, but I think it helps.</li>
<li>Your underlying system is written in C and many common system software
based applications are written in C.</li>
<li>Being able to patch systems, find bugs, root cause things to source code
is jedi type stuff. Strong C skills give you confidence to do things
around the system you won’t get elsewhere.</li>
</ul>
</li>
<li>Learn a version control system
<ul>
<li>If you’re not sure, use git.</li>
</ul>
</li>
<li>Learn how to develop and use HTTP based API’s.
<ul>
<li>Interact with AWS to spin up VM’s on demand</li>
</ul>
</li>
<li>Read and write lots of code!!
<ul>
<li>Pick up simpler systems - I often look at the early versions of systems
before they got complicated. First version of git. First version of nginx.
First version of Openstack</li>
</ul>
</li>
</ul>
<h2 id="internet--networked-environment-focus">Internet / networked environment focus</h2>
<ul>
<li>Understand common protocols and know how to debug them
<ul>
<li>SMTP - You should be able to execute an SMTP transaction by hand :-)</li>
<li>DNS - be intimately familiar with dig and how DNS requests work.</li>
<li>HTTP - be able to trace HTTP requests.</li>
<li>TCP/IP - there’s so much to learn here, but learn to make tcpdump your
friend</li>
</ul>
</li>
<li>Encryption
<ul>
<li>Understand PKI and learn how to use GPG</li>
<li>Understand one-way hashing </li>
</ul>
</li>
<li>Understand time
<ul>
<li>What UTC is</li>
<li>What is NTP and how does it work?</li>
</ul>
</li>
<li>Learn LDAP
<ul>
<li>LDAP’s a pain, but still very prevalent. Give OpenLDAP a whirl.</li>
</ul>
</li>
<li>Learn load balancing
<ul>
<li>Use HAproxy to distribute load to a bunch of webservers.</li>
</ul>
</li>
<li>Learn caching
<ul>
<li>Learn Varnish and the benefits to a caching layer </li>
</ul>
</li>
<li>Learn Redis
<ul>
<li>A simple, very popular key value store. Write some code that uses Redis.</li>
</ul>
</li>
</ul>
<h2 id="putting-it-all-together---practical-examples">Putting it all together - practical examples</h2>
<ul>
<li>Write a small web application using whatever langauge or framework you know.
Make sure it uses an RDBMS (e.g. Postgres/MySQL). Write some code to spin up a
VM on AWS and deploy your application using your config mgmt system of choice.</li>
<li>Setup an email address on your local email application that when you send it
an email with a secret key can add an entry in to your DNS system.</li>
<li>Write code to be able to add and remove users / groups from an LDAP system
which is responsible for authenticating a whole series of systems.</li>
<li>Build a system which automatically detects when your webserver is down and
automatically starts it back up again when it does.</li>
<li>Generate load on your webserver and tune / optimize it so that it can handle
way more than what it does today.</li>
</ul>
<h2 id="and">… and</h2>
<p>… here’s the bad news. You’re probably still not there. The next stage is
building these systems at scale and seeing where they break and knowing
where to put effort in to making them robust and reliable.</p>
<p>Here’s the good news though. You have more of a foundation of knowledge than
90% of the folks out there practicing devops and you have a fantastic base to
build from.</p>
<h2 id="resources">Resources</h2>
<p>Read, read, read. Spend your spare time reading.</p>
<ul>
<li>Google is your friend obviously. You should be able to find oodles of
tutorials online</li>
<li>Usenet - there’s still a lot of activity online around Usenet and Usenet
groups.</li>
<li>Mailing lists for popular / large pieces of software (e.g. Postfix / nginx).
I used to read a couple of hundred posts per day from these and you just pick
up things through osmosis.</li>
<li>Stackexchange - go figure out how to answer questions on serverfault.</li>
<li>Reddit - lots of subs related to Linux.</li>
</ul>
<p><a href="https://brassy.net/articles/getting-in-to-devops">I want to be a Devops!</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on November 24, 2015.</p>https://brassy.net/articles/rules-for-delivery2015-11-23T00:00:00+00:002015-11-23T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<p>These are rules I would love to enforce for any 3rd party application vendor
when you’re shipping stuff to me. I love the <a href="http://12factor.net/">12 Factor</a>
rules but they are lengthy. I get most of the value from these… </p>
<h3 id="versioning">Versioning</h3>
<ul>
<li>Version all artifacts</li>
<li>Use <a href="http://semver.org/">Semantic Versioning</a></li>
<li>Artifacts are sacrosanct - once an artifact is produced at a specific
version, it is essentially a shippable product. Don’t open it up to change
something. Need to fix something? Revision and fix forward.</li>
</ul>
<h3 id="packaging">Packaging</h3>
<ul>
<li>Package using a non-distribution & non-platform specific package format
first (e.g. .zip/.tar.gz) Then build distribution / platform specific
packages.</li>
<li>If you’re using a binary packaging model (e.g. rpm), make your source
available for packaging.</li>
<li>Include the version in the package name Don’t package up third party open
source tools and deliver them on someone else’s behalf.</li>
<li>List dependencies in the packages. If possible, don’t tie to specific minor
versions of tools. i.e. depend on MySQL 5.1 not MySQL 5.1.73</li>
<li>Include documentation in your package about installing & configuring your
software. A README.txt or README.md is pretty normal. </li>
<li>Please do not add do a bunch of pre or post install magic in your packaging upon
installation. Copying files to directories, setting up permissions, relevant
users etc. is all fine. </li>
</ul>
<h3 id="installation">Installation</h3>
<ul>
<li>Where using distribution specific packages, a simple yum install
top-level-package should install your package and all of it’s dependencies.</li>
<li>Please try and adhere to <a href="http://www.pathname.com/fhs/">FHS</a> and the
distributions standards for installation on *nix operating systems. </li>
<li>Please integrate in with the local system startup for the distribution you’re
targeting when providing distribution specific packages.</li>
<li>Install with sane defaults that preferably allow me just to start your
application without configuring stuff, or with minimal (obvious)
configuration. Helps me play with stuff in dev.</li>
</ul>
<h3 id="support--production">Support / Production</h3>
<ul>
<li>LTS - Have a support strategy that lines in with common distribution LTS
support strategies.</li>
<li>Have a debug mode which enables more verbose logging and allows you to
capture inputs & outputs from your various channels/services. Particularly
in a distributed application. </li>
<li>If configuration is complicated (i.e. you need to edit more than one file),
provide a way to show configuration </li>
</ul>
<p><em>With the amount of code, libraries and full blown systems now out there,
delivery is becoming more important. Don’t try and re-invent the wheel. Plenty
of other people have been doing this for some time.</em></p>
<p><a href="https://brassy.net/articles/rules-for-delivery">Rules for Software Delivery ...</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on November 23, 2015.</p>https://brassy.net/articles/running-cross-functional-service-teams2015-08-01T00:00:00-00:002015-08-01T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<h2 id="preamble">Preamble</h2>
<p>This is, in written form, <a href="http://velocityconf.com/velocityeu2014/public/schedule/detail/37211">something I talked about at Velocity Europe
2014</a>.
Based on feedback from the talk, I’ve decided a blog post might be a nice way
for people to refer back to this.</p>
<p>The idea of cross functional teams is that the team can be autonomous to do
what it needs to do to develop, build, test, qa, deploy, release, monitor &
maintain it’s product or service. This means one team which has the entire
skillset required to own the <em>full lifecycle</em> of it’s deliverable. You need to
either hire or grow a skillset that is often foreign to traditional development
teams with respect to areas like deployment & monitoring. </p>
<p>From a technical perspective, if you’ve got a small product (like a small LAMP
stack in a startup), then this can be straight-forward. As your architecture
grows, you need to figure how to compose your application so it becomes more
manageable and maintainable. One popular way of doing this is breaking down
your application in to services ala <a href="http://en.wikipedia.org/wiki/Service-oriented_architecture">a service orientated
architecture</a>. </p>
<p>The <a href="http://martinfowler.com/articles/microservices.html">topic of
microservices</a> has been
particularly prominent recently and the principles here align very nicely with
microservices if that is how you choose to grow your application. </p>
<p>I’ll continue to use the word “service” as the item to organise around. If
you’re small, that service may be your entire application.</p>
<p>This idea is particularly suited towards cloud software where you run your own
infrastructure. If you are shipping mobile software to a 3rd party device, or
selling on-premise software to your customers this article may not as relevant
to you. </p>
<p>Typically, the three main protagonists in any cloud software company for
building and running the software are Developers, QA & Operations.
Increasingly we start to see Security (SecOps / AppSec / Infosec) as part of
this story also. Historically, those organisations have been siloed with their
own reporting and management chains. Developers would develop stuff, often
times working closely with QA (or not!) and then the proverbial “throw it over
the wall” moment would happen as they deliver to Operations. However there has
been a tremendous push recently towards
<a href="http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/">“DevOps”</a>
led by Patrick Debois. It will be difficult for me to write this without
repeating some of the DevOps mantra, but I’ll do my best to try and focus on
the specifics of running a cross functional team. Most of the DevOps
principles are very abstract and I hope to espouse some more of the concrete
benefits I’ve seen.</p>
<p>As we develop more rapidly, deploy more often and maintain ever larger pieces
of infrastructure our inefficiencies become exacerbated. The goal we are striving
towards is the ability for the team who is building a service to own their own
destiny and be autonmous where they need to be.</p>
<h2 id="whats-wrong-with-the-traditional-model">What’s wrong with the traditional model?</h2>
<p>Firstly, most of what I’m defining refers to a specific landscape. These
symptoms appear in cloud software companies who have multiple development
groups producing part of the product(s) that typically lives on an ever
growing and more complex infrastructure. Separately, you have an operations
group which is responsible for running the stuff the developers have built.
They’re often also responsible for building the stuff that the stuff the
developers built runs on.</p>
<p>Over time your operations group grows bigger and bigger. With a bigger
application their application knowledge spreads thin. You’ve introduced a
throw it over the wall mentality where developers introduce new things that
are decided on in a sprint that they don’t understand the full consequences of
in production. You’ve got lack-lustre monitoring with an ops team clamouring
to figure out what to monitor where. Typically after it goes in to production.
Your development teams & managers need qualifications in project management to
now manage all of the cross team dependencies when they have new stuff to
deploy or have to make significant changes, particularly where there’s
deployment impact. Those cross team dependencies often manifest as meetings
and as a development manager you’ll find yourself in meetings constantly with
managers from other groups talking about what is it you’re putting in to
production and how that works.</p>
<p>The key metric for cloud software is <em>delivery</em>. What do we mean by delivery?
Running in production. You’ve given developers autonomy over everything that
happens before they deliver (build, test etc.), but their delivery channel is
non-existant or extremely manual. Delivery becomes a path fraught with peril. </p>
<p>Another interesting part of this is scaling the organisation. As the
development & operations organisations grow, resourcing becomes much more
tricky. The development organisation often grow independently from the
operations organisation. In theory, both could equally outgrow the other but.
All anecdotal evidence seems to point to operations as a bottleneck in most
companies as opposed to development. It feels like that should only happen
some of the time given the law of averages, so something else is at play here.
Seems like we’re playing with a stacked deck. Stacked against operations.</p>
<p>New folks in operations start to have increasingly less familiarity with the
systems. They don’t have all the tribal knowledge of the older folks and as the
system grows it becomes increasingly difficult to understand all of the moving
parts together. </p>
<p>The siloing of the teams means often developers and operations folks only ever
talk when something new is about to get in to production or something is on
fire. Building a relationship in emergencies is certainly a bonding
expeirence, but then the only way those relationships get built is if you have
a lot of fires. Don’t think that’s something we want to encourage.</p>
<p>Simply, <em>done isn’t done until it’s delivered</em>. If a team can’t own ‘done’,
then they don’t own their service. You now have a shared ownership model
between operations and development and to coin a cliché, “if everyone owns it,
no one owns it”. This manifests itself in many many ways, most of which are
covered more appropiately in other devopsy related topics you can read
elsewhere.</p>
<h2 id="embed-in-the-scrum-team">Embed in the scrum team</h2>
<p>“<em>You build it, you run it</em>” was the phrase coined by <a href="http://www.allthingsdistributed.com/">Werner
Vogels</a> to describe Amazon’s model. This
has more benefits than just empowering the development team. It starts to make
operational issues front and center. It forces them to understand their service
running in production more but likewise that allows them to make better
decisions. </p>
<p><a href="http://en.wikipedia.org/wiki/Principle_of_least_effort">The principle of least
effort</a> suggests that
once you’ve established an organisation that has now introduced significant
friction in to the delivery process, that your developers, who are trying to
deliver, will often choose paths that reduce their friction even if that’s not
the best path. That can often mean making bad architectural choices rather
than going through the slog of introducing more artifacts which complicates
deployment and makes them end up in meetings. Suddenly, adding something new
to the mix becomes a tradeoff discussion as you talk about the logistics of
co-ordinating that. </p>
<p>So, here’s my proposal. Embed all of the knowledge required in the scrum team.
Set that as a goal. The scrum team needs to own all aspects of their service.</p>
<p>Delivery is a feature.
Upgrade is a feature.
Performance is a feature.
Monitoring is a feature.
Uptime is a feature.</p>
<p>If you do not own those areas in your scrum, then those features are owned
outside of your scrum team. You’ve now created dependencies. Uh oh.</p>
<h2 id="dependency-vs-redundancy">Dependency vs Redundancy</h2>
<p>There is a natural inclination for every engineer to try to re-use as much as
possible. Everything we’re taught in software engineering lends itself to
re-using code. Well written, well designed code following <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY
principles</a>. </p>
<p>The engineering groups natural inclination is, <em>“Shouldn’t a build team take
care of our builds? Shouldn’t a deployment team take care of our deployments?
Shouldn’t a monitoring team take care of our monitoring?”</em>. We then centralize
those resources. Organisational efficiency at it’s finest. Those are
specialized skills that are consolidated within those groups. Easier to hire
for. Easier to manage. Dedicated manager. On-paper perfection. </p>
<p>Unfortunately, reality bites and, in most cases, no-one really understands
what we’ve created from a project delivery standpoint. First of all, you need
to accept that your <em>definition of done, is delivered</em>. Now start to draw the
dependency tree of teams that it takes to deliver small features & changes for
just one engineering team & you start to see the challenges. That dependency
diagram continues to get more and more complicated as the organisation grows.
Typically those same groups have multiple other groups that depend on them
also. Once you end up in this situation you end up in resource contention
hell. A lot of developers can actually relate to this and have experienced
this intimately in their job. It’s called <a href="https://en.wikipedia.org/wiki/Dependency_hell">Dependency
Hell</a>. Who knew, it existed
within organisations too! </p>
<p>So, the alternative to dependency is redundancy. We end up with multiple teams
potentially inventing the same solution to a similar-sounding problem (e.g.
caching services, queueing services). So, the idea is not to deal with
redundancy up-front by trying to eliminate everywhere there is redudancy. The
idea is, to bludgeon more development metaphors, refactor as and when you see
it.</p>
<p>Embrace redundancy! Then figure out how to remove it long term. Don’t make
removing redundant services or functions dependencies on delivering. Embrace,
potentially, reinventing a slightly less-round wheel. You can get to make it
rounder over time.</p>
<p>So, hold on a minute, you say, if we brought this thought to a logical
conclusion & every team did everything themselves, they’d run their own IT
infrastructure, build data centers, build their own laptops. That’s obviously
not a runner. How do we know when to choose redundancy over dependency? Simple
answer… when it’s a problem. When it’s stopping you getting things done.</p>
<p><em>The goal should be to minimize as many external dependencies as possible.</em></p>
<h2 id="organisational-growth">Organisational Growth</h2>
<p>One of the side benefits to traditional development teams now owning more of
their delivery pipeline is headcount planning. </p>
<p>Often, resources that own different parts of the delivery chain live in
different parts of the organisation. When that happens you now have
complicated resourcing strategies as different parts of the organisation grow.
Inevitably, you end up with capacity constraints somewhere and inevitably
those constraints end up on the teams that are the most contended i.e. have
the most dependencies. </p>
<p>Owning your build & delivery chain and minimizing those external dependencies
makes headcount planning a lot easier. The dev team owns the headcount, one
manager makes a decision and a business case. The decision therefore can be
made at the closest point to the actual resourcing problem - at the team
level. </p>
<h2 id="the-people">The people</h2>
<p>The last set of benefits I’ll talk about is on the people side. </p>
<p>The first and most important trait of having all of this knowledge on one team
is growing a deeper understanding of the system with your engineers. If your
delivery chain, build pipeline or your infrastructure is complicated and/or
nuanced this becomes more important. Having the engineers understand these
areas allows them to dive deep in the belly of the beast, where necessary.
Even if you only have a few people who can do that, that knowledge tends to
spread when it’s <em>within the team</em>.</p>
<p>Second on my list is autonomy. Suddenly, when you own more of the system, you
are free to make different choices. You genuinely become more agile. If you
need to upgrade to a newer version of your language framework, test harnass
etc. you can do that. Turning on a dime is difficult in any software company,
but owning more of your process is the key to make sure you’re not trying to
turn the titanic. Co-ordination with other groups is difficult. One of the
consequences is that you find over time, people hesitate to suggest things
that require large amounts of effort. Now you’re free to try new things, to
experiment and iterate towards better solutions for your builds, delivery
pipeline etc. When small changes take large amounts of co-ordination and
logistics, iterating towards a solution is always painful & risky. This type
of perceived red-tape actually starts to penetrate developers well-being.
Moaning & griping can often happen as teams complain about how long simple
things take to do. Let them own it.</p>
<p>Last is cross training. You’ll find that having e.g. ops and dev folks on the
same team means knowledge starts to filtrate between the two groups of people.
The developers and ops folks often end up blending in to the same role where
it can often become difficult on the surface to even know you have different
skillsets within the group.</p>
<p><em>As we scale, we figure out new and imperfect practices that hamper our
ability to grow & sell product. Sometimes these changes are fundamentally
different ways to execute and require huge buy-in. Don’t underestimate the sea
of change within your organisation required to make this happen.</em></p>
<p><a href="https://brassy.net/articles/running-cross-functional-service-teams">Running Cross Functional Service Teams</a> was originally published by Philip Reynolds at <a href="https://brassy.net">Philip Reynolds</a> on August 01, 2015.</p>https://brassy.net/articles/cidr-and-basic-subnetting2015-01-01T00:00:00-00:002007-03-01T00:00:00+00:00Philip Reynoldshttps://brassy.netphilip.reynolds@gmail.com"<p>This particular aspect of networking/computer affects everyone with a network.
It’s so unbelievably necessary for anyone who deals with networks (at any
level) to understand. I cannot stress this enough! :) That said, so many
people have serious problems with it and I can kind of understand why. Every
single site I’ve seen goes into the detail straight away with binary
arithmetic. Yes, binary arithmetic is how you work these things out, but are
these people doing this binary arithmetic in their head when they know a /26
has a subnet mask of 255.255.255.192? Well, yes and no. They’re using a
shorthand variety, they are not ANDing 1 and 0’s. Well, if they are, they’re
wasting braincells.</p>
<p>Do you even know what I just said? No? Well read on… Warning : There’s very
little point skimming this. It might be time to open up your calculator and
grab a piece of paper and a cup of coke (ugh, I hate coffee).</p>
<p>So, a computer network is basically just a way for my computer to talk to
another computer. If you need this part explained, it might be time to start
reading somewhere else before continuing here. So on our IP based networks,
machines have IP addresses. So, yadda yadda yadda, this uniquely identifies a
machine on the network. I’m hoping noone needed that to be explained. If so,
go read some background first. Of course, machines can have multiple IP
addresses and even one IP address can represent multiple machines, but let’s
keep it simple for the time being (if you don’t put these things in, the
anoraks get very animated).</p>
<p>So back in the good ol’ days, when networks were easy we had what is known as
… Park Life … ok, sorry, bad habit … classful networks. It’s surprising
how many texts and papers (mostly out of date) still refer to Class A, B & C
networks. These are obsolete and have been since about 1993 (ref: wikipedia -
it’s great isn’t it?). So basically, these classful networks, i.e. what is
commonly referred to as Class A, B & C have been out of date since anyone
actually used the Internet. What the hell is up with people still referring to
them then? I don’t know either. What is used now is CIDR - Classless
Inter-Domain Routing. Don’t get too bogged down remembering it, just associate
network segments with the slash terminology - /16, /24, /28 (pronounced “slash
16”, “slash 24”, “slash 28”) etc.</p>
<p>So what were classful networks about and why do we now have CIDR? Well, a
network is great, but although we have one big network (the Internet) it’s
actually divided up into many small segments. This is to facilitate many
things. There have probably been many papers written on the benefits and right
and wrong ways to segment networks but at the end of the day it makes logical
sense that in some way the big network containing 2^32 IP addresses is broken
up into smaller chunks that are more manageable. In the before time (i.e.
before 1993) you (as a person who wanted an allocation of public IP addresses)
could have three types of allocations. A class A (big allocation - like really
big), a class B (medium sized) and a class C (small sized). These were fixed
sizes, so a Class A was 16777216 (or 2^24 - that’s 2 to the power of 24) IP
addresses, a class B was 65536 IP addresses (i.e. 2^16) and a class C was 256
IP addresses (i.e. 2^8).</p>
<p>Now, what about if someone wanted something in between a class C and a Class B
(like say 20,000 ip addresses) ? Well, they’d either have to get allocated
approximately 78 class C’s (20,000 divided by 256) or use a class B and waste
40k IP addresses. What if someone wanted 12 IP addresses? Well, they’d get a
class C as well … eek. Ok, so we moved away from this technique to stop
WASTING IP addresses. Why don’t we hand out IP addresses one at a time?
Because that isn’t feasible & it makes sense to use logical blocks for a lot
of reasons but especially routing ones. There’s a long winded explanation for
that as well, so just ignore that question! </p>
<p>So, the CIDR format was used in order to facilitate chopping up network
segments into smaller more configurable chunks.
Now, this is the bit where we will need to start using some binary arithmetic,
but don’t worry, I’ll try and guide you through it.</p>
<p>First and foremost:
An IP address is 32 bits. In fact in networking, most things are represented
in 32 bits as most computers can natively (or at least were, we’re moving
towards 64 bit architectures now) store 32 bits. What does that mean? It means
that the IP address 89.234.64.66 (the IP of this website) is stored as 32 1’s
and 0’s.</p>
<p>IP addresses are composed of four parts (octets), so if an IP address is 32
bits, each octet is how many bits? (octet should give you a clue as well) …
yup, ok it’s 8 (again, simple arithmetic, 32 divided by 4).</p>
<p>So, in 1’s and 0’s, the IP is broken up into:</p>
<div class="highlight"><pre><code class="language-c" data-lang="c"><span class="err">