Handy util when writing Python code that includes multiline strings: textwrap.dedent
Month: September 2014
The Twelve-Factor App by Kristian Glass
Notes taken during a talk at PyConUK
#1 one code repo many deploys
#2 what about declaring your dependencies, e.g. gcc compiled items, deply containers
#3 read config from environment
debug flag, db connections etc
avoids secret leakage,
add defaults
make it language independent
populating env – foreman, supervisor (d?), ansible, bash -whatever etc
doismellburning – django app
www.doismellburning. co.uk .com?
NB: can be in another repo, just def outside of your app
#4 Loose coupling
#5 build, release (when config gets blended), run
build tip
…
fpm – builds an os package (ruby, don’t snigger too much – it’s awesome)
run stage needs to be a simple as possibly, do as much work in advance. e.g. fetch eggs in build, collect static etc
#6 stateless
assume nothing about RAM, persist data to external serices
#7 Port binding:
clean interface, should be able to tell your app a protocol and port and go
#8 Scale out not up, start more processes, don’t daemonise – let the env do it
#9 Stop quick;y, stop nicely
#10 Keep env similar: time, people, tool gap
Deploy more often
people
-> get people together, devops, …
tools
-> it works on my machine
#11 Treat logs as event streams
Run a log agreegator
one event, one line
=> q. how do you handle exceptions – let the agreegotr handle this, some differently code new lines.
#12 Run admin tasks in your app, eg clean up old sessions, as a django management command not a one off script on some dev machine
Language and framework agnostics
Performance
Notes taken during a Performance talk given at PyConUK.
function calls
c method instead
cython instead
Use operators
Loops
Avoid
list comprehension is quicker than for loop over an emnumerator
What is op?
xrange vs range
– no wasted memory
NB: In python3 renamed range
itertools
Or use a c extension
mstexttools
Sorting
l.sort vs sorted() – any iterator
Key based:
look up operator -> operator.itemgetter
Can use on object attribute
Strings:
“”.join([a, b, c]) faster but less readable
% method not that fast an ugly – don’t use
Dictionary
Int keys are faster than string keys
interned string keys – no garbage collection
sys.intern – P3
Variable look ups
Mentioned global variable trick
local var faster than global in general
Look up: Old style/new style classes??
Slot attributes ?? look up more just seemed to be attributes???
Don’t use exceptions as an often use case. => much faster say to use getattr than catch a valueexception
Try to rely on c methods
Profiling tools:
c profiler built into Python
cpython vs number – can make pythonrun as fast as C – suggestion
A Pinch of Salt by Scott Walton
Scott Walton gave a talk about how they use Salt, at their company Pebble.
Slides can be found here: https://docs.google.com/presentation/d/1q82GUdE-zHUlTSYJxkIdW79CcewO3-KXuBwPe9ir_aA/edit#slide=id.p
Salt is a remote config management tool, similar to: Ansible, Puppet, Chef. It seems from talks with others at PyConUK Ansible and Salt were being described more favourably than Puppet and Chef. Either due to their complexity or through battle testing and reaching their limitations.
Salt terminology
- Master
- Minion
- Grain – data on a minion – like an attribute, how much memory (-G)
- Pillar – data about a set of a minion, e.g. stage of server
- Matching – rules to identify minions
- Modules – commands
- State – end result
Pebble use Salt to manage a complex system, finance, web servers, databases, load balancing, managing various environments – test, stage, prod etc.
Their journey was via fabric
Implemented in Python
Zero MQ, encrypted secure connections are used
Used apt-get vs pip to install – as in his opinion, it included a flaky dependency
Can add watches for changes to take actions, e.g. if web server config file changes, restart the web server
Can use Jinga in config files pillar (and grain) are available in these templates
Salt can take fall back actions for common features, e.g. running git clone or git pull depending upon whether the repo exists.
– started using 2 years ago
Comparisons:
– chef & puppet looked more complicated
– salt looked for familiar
– friendly community
– easy to get involved in the code
– can easily contribute back pull requests
– documentation has gotten better
– includes very simple state
– reference is very good
– e.g. ‘return as’ is not as well documented as yet
Easy to configure
can command Less files TBC
If not built in, can just run bash commands e.g. could run a Django migration
Trouble at t’LeedsDataMill: Oompah.py ~ Big Data meets Big Brass (Nicholas Tollervey & Simon Davy)
Trouble at t’LeedsDataMill: Oompah.py ~ Big Data meets Big Brass (Nicholas Tollervey & Simon Davy)
Leeds data mill – hackers of open data
Played with footfall data represented as music
Python created a Lilypod file –
=> 7am was a fav Tubor section
Much jokes of a Dubstep/D&B version – When’s the next album out – lol