Dmytro Kovalov: Personal projects, blog, tips and tricks, UNIX development, administration and devops.

iPhoto and files permission

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 16 Sep 2014 00:00:00 +0900

Why is it so difficult to execute chown or chmod with root password? Migrating photo album from iPhoto 8 to iPhoto 9 does not seem and easy task on Mac anymore.

First of all iPhoto refuses to upgrade any old library, if it resides on Netatalk volume. It just hangs indefinitely without seemingly doing anything. Then, after you copy all you library to local disk and start rebuilding/upgrading you'd see following dialogs:

But... I'm sorry, iPhoto! I just gave you my root password!? What else do you need to repair permissions? Why simple chmod -R 777 works for me and not for you?

Worth mentioning that upgrades are not 100% safe, some libraries are partially broken after completion... :(

Is MacOSX 10.9 == MacOS 9.x?

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 20 Dec 2013 00:00:00 +0900

My Mac is crashing... Again, after 10+ years of stability. . I am mac user since MacOS 7.1. 7.x never crashed on me, 8.x was kind of OK, 9.x sucked. .

Now after upgrading to Maverick I am rebooting more or less regularly to fix some of the problems. I never had any problems on any of the Macs with audio or with camera (on Macs with cameras). Currently I have already rebooted several times to fix missing sound, and my camera in Skype still does not work.

As a UNIX SA I know for sure that nobody needs to reboot UNIX based computer for any reason but kernel upgrade. Not if you are Mac user anymore...

</end_of_grunt>

Same sad story with iOS 7. It sucks seriously. Because of permanent crashing and all other bugs. This is the worst iOS version I've used.

Apple problems started last time when Steve Jobs was kicked out of Apple. Then he was back, Mac was back. Now, sadly, there's no way to get him again. Does it mean Apple is going down forever? Hope not so.

Running Chef roles from Capistrano

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 04 Dec 2013 00:00:00 +0900

Chef databags are used to pass information between sessions of Capistrano and Chef. Databags are generated on Capistrano side and then used by Chef for server and components configuration.

In my new project und Capiche I use tandem of Capistrano (or cap) and Chef-solo (chef) to deploy stacks of applications. Stacks are collections of server configuration, software packages required to run application, software configuration, and application code itself.

Capistrano + Chef

Here's a brief flow of typical deployment process in this schema:

locally capistrano creates configuration to be used in Chef;
cap copies Chef-solo and configuration to all remote hosts;
cap starts chef-solo on remote hosts;
after chef finished, cap starts standard application deployment.

Databag messenger

There are significant differences in how cap and chef executed. First and main difference is that they are invoked on different servers. Cap starts deployment from local server, but chef processes (possibly many of them) run on remote servers in parallel.

There are no standard ways (like calling method and passing arguments) for information exchange between cap and chef. I use chef databags as information medium between cap and chef.

Capistrano needs an ability to generate databags from its own configuration in the format that chef cookbooks understand; also capistrano should be able to read and search chef databags.

Hosts (AKA nodes) databag

First and very important task, is to be able to pass server configuration -- including roles -- from local Capistrano process to the remote nodes, where chef can use it. Example below shows how this could be implemented.

Suppose you have hosts configuration like the following:

server '10.0.1.10',  :app, :web, :db, :admin, hostname: "mysql01", primary: true
server '10.0.1.11',  :app, :web,              hostname: "web01"
server '10.0.1.12',  :app, :web,              hostname: "web02"
server '10.0.1.13', :logger, :dns, :security, :monitoring, no_release: true, hostname: "master"

From this configuration Capistrano recipe creates databag called :node with one item for each capistrano host. Example of a recipe:

task :roles do
    find_servers.each do |server|
        File.open("#{dir}/#{server}.json", "w") do |f|
            f.print({
                id:        server.host.gsub(/\./,'_'),
                role:      role_names_for_host(server),
                fqdn:      server.options[:hostname] || server.host, 
                ipaddress: server.host,
                options:   server.options
            }.to_json)
            f.close
        end
    end
end

Generated databag for one of the host looks like:

{
    "id"        : "10_0_1_11",
    "role"      : ["logger", "dns", "security", "monitoring"],
    "fqdn"      : "10.0.1.11",
    "ipdaress"   : "10.0.1.11",
     "options" : {
         "no_release"    : "true",
         "hostname"      : "master"
    }
}

This databag can be used directly with chef recipes, using data_bag_item or search methods as for example:

servers    = search(:node, '*')
monitoring = search(:node, "roles:*monitoring*")

Note: There is one trick though. Recipe needs to be written in such a way as to support search in Chef-solo.

By default Chef-solo is only able to use data bags for data_bag and data_bag_item operations; to use search with solo you need to use extension chef-solo-search.

Note 2: In some cases this is not enough, however: if cookbook designed in such a way that search is explicitly prohibited in solo mode, some changes to the cookbook are necessary. Below is an example of the modifications I had to do for Munin cookbook to make it work with both Chef-solo and search.

# Original cookbook recipe
if Chef::Config[:solo]
   sysadmins = data_bag('users').map { |user| data_bag_item('users', user) }

# Change to
cant_search = Chef::Config[:solo] && node.recipes.grep(/chef-solo-search/).empty?
if cant_search
   sysadmins = data_bag('users').map { |user| data_bag_item('users', user) }

Applying Chef roles

There are two sides to actual executing a Chef cookbook via capistrano: cap side and chef. On every server we need to execute command that applies configuration specific to that server. We could do this sequentially, by using :hosts option to run method, but this will be really slow.

Instead of this, I am using a bit of Capistrano magic and small Ruby script on remote server(s) side.

Capistrano task

Following task starts parallel run of remote script run_roles.rb on all servers with option that contains server name from cap configuration. This $CAPISTRANO:HOST$ variable is special Capistrano magic, it is internally converted by Capistrano to server.host (Note: It is not shell variable, it is set by Capistrano before shell runs).

task :roles do 
    run %Q{ #{try_sudo} #{chef_solo_remote}/run_roles.rb  $CAPISTRANO:HOST$ }
end

On remote (i.e. chef side) there's a small script, that takes server.host as an argument and applies roles from node databag.

What happens here is the following:

script reads databag with current server configuration;
it reads all roles files (role.json) that are listed in the databag;
combines them into single run_list; and
runs chef-solo using combined run list.

Chef::Config[:solo] = true
Chef::Config[:data_bag_path] = "#{current_path}/data_bags"
current_host = ARGV[0].gsub(/\./,'_')

exit unless current_host

roles =  Chef::DataBagItem.load(:node, current_host)["role"]

run_lists = []
roles.map{ |x| "#{ current_path}/#{x}.json"}.each do |role|
  run_lists += JSON.parse(File.read role)["run_list"]
end

File.open("#{current_path}/#{current_host}.json", "w") do |f|
  f.print({ run_list: run_lists.compact.uniq }.to_json)
end

cmd = "cd #{current_path} && chef-solo --config solo.rb --json-attributes #{current_host}.json"
begin
  PTY.spawn (cmd) do |stdin, stout, pid|
    begin
      stdin.each { |line| print line }
    rescue Errno::EIO
    end
  end
rescue PTY::ChildExited
  puts "The child process exited!"
end

Next steps

To be able to deploy full stacks of applications additionally to basic hosts information Capistrano needs to share all its configuration with Chef. In next installments I will describe how this can be achieved.

und Capiche

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 01 Nov 2013 00:00:00 +0900

What is it?

Capistrano is primarily used as web application deployment framework. It is excellent combination of remote parallel execution of SSH commands with work-flow management. Chef, on the other side, is designed as configuration management tool and mostly used to configure infrastructure. Is it possible to combine powers of Capistrano and Chef for simpler and faster deployments, with simplified configuration? Can I deploy infra and application in a single step?

I've made several attempts to combine powers of Chef and Capistrano: aws_deploy, chef-solo, capistrano-recipes. This is a start for the new project.

Application Centric Architectures

In the old days deploying a server took longer. Rack it: optimistically - one day (if you're lucky, you could reuse somebody else's retired box); you (or your SA's) install OS on it - another day (with all the approvals and manual configs). In extreme cases (not a joke!) deploying a server took me 5 to 6 months.

Application design was largely influenced or dictated by infrastructure architecture. Not only because of the price of the hardware, but also because of the difficulties with the deployment, multiple applications were sharing single server: "Our existing DB server has a lot of RAM and disks, we will put every possible database on it. When we run out, we will add disks and memory or migrate box to larger hardware or we build a cluster."

These days architecture of the infrastructure is rather influenced by the application environment. Each application can have distinct architecture, different from any other application or even dynamically changing architecture, depending on the needs. Since deploy of a new server can take less than a minute, they are created when required, and there is no need to share.

Management

Dynamics of the environment makes administration of the architecture easier and more difficult at the same time.

Easier, because of the one function per server approach. We can decommission (or upgrade) server and be sure that it did not serve any additional tasks that we are not aware of. For exactly the same reason shorter life-time of the server(s) is better.

On the other side, management is a bit more difficult, because of higher number of servers that you need to manage. But this is where devops tools can really help.

What is my goal for this project?

I want to build servers from scratch. If not from bare metal, then at least from 'bare OS installation'. Not just servers, but 'architectures' - tailored for applications. Servers organized into logical groups, with application functioning at the end of single step deployment.

I want to:

have a list of hosts or IP's;
an SSH key(s);
Sudo or su access;
define server role(s), like :app (Note: I am following Ruby and Capistrano conventions here), :monitoring, :logger;
run deployment;
have running application on newly deployed server(s) is XX minutes;

I want to avoid:

This should be easy to configure for end user, avoid writing extensive configuration for each new application or 'architecture': just hostnames and roles.
Additionally, I want to be independent from any specifics of cloud or non-cloud provider.
Similarly, I wish avoid building custom configuration and deployment management infrastructure (such as Chef or Puppet servers). My idea is to bring my laptop to you, plug it in (or let you git clone configuration repository), and by running single command have your application deployed in minutes (or, let's say dozens of minutes).

Yes, there are drawbacks. Configuring lists of hostnames or IP addresses in files is not scalable in the long run -- for large-scale applications or for multiple applications, for hundreds of hosts. One step at a time...

Chef and Capistrano

I had made several approaches trying to make Cap and Chef work together.

First attempt was a project called aws_deploy. Simply speaking it is a Ruby wrapper around AWS API, Chef-solo and Capistrano. Goal of the project was to have a single step deployment of application server in EC2 environment. AWS deploy is able to spawn new AWS instance if it does not exist, bootstrap it and install required git branch of RoR application.

Although this project simplified deployment process a bit, shortcomings of the AWS Deploy design are obvious:

AWS Deploy is able to deploy only single instance at a time;
chef-solo is not really configurable and only delivers standard static configuration server prepared to run Ruby on Rails;
there is no interaction between Capistrano and Chef, they simply executed one after another by script wrapper.

Cap + Chef = Capiche

Capistrano recipes often have some kind of installation task. Before deploying a configuration, you have to install the package. Such recipes are usually similar to the example below:

namespace :nginx do
    task :install do
        sudo "apt-get install -y nginx"
    end
end

There's an obvious problem with the recipe: it is not portable. This will work only on Debian or Ubuntu, and even on these systems there cases when this is not enough: what if I want different version of Nginx, or if I want to compile it from source with some custom configuration.

At the same time Chef is designed to handle such situations, and have all necessary DSL for it.

Is it possible to change Capistrano recipe like one above and make it use Chef cookbook instead?

This is exactly what I am trying to achieve with the new project I am calling Unified Deployment with Capistrano and Chef or 'und Capiche' (AKA Capiche) for short.

The new project is not really new, it's rather a combination of the efforts from previous similar projects. Some things is already implemented and working, as I write this. But functionality and code are spread across multiple repositories and it's not easy even for myself to join pieces together to design deployment for new project.

Components

Current Capiche setup is based on RVM, Capistrano 2.x, Chef-solo and Capistrano multi-configuration extension.

Current state

Below is an example of how configuration file for an application. This is actual working setup.

server '10.0.1.10',  :app, :web, :db, :admin, hostname: "mysql01", primary: true
server '10.0.1.11',  :app, :web,              hostname: "web01"
server '10.0.1.12',  :app, :web,              hostname: "web02"

server '10.0.1.13', :logger, :dns, :security, :monitoring, no_release: true, hostname: "master"

:app, :web and :db are standard Capistrano roles for Rails application,
additional roles present here are :admin, :logger, :security, and :monitoring - new roles can be defined as necessary.
Similarly :primary and :no_release are standard Capistrano host options, but
:hostname is custom option used to set target host hostname and DNS.

Capistrano roles one-to-one correspond to roles on Chef side. This allows skip additional configuration.

Each Chef JSON role is simply a pointer to a more details role Ruby file. JSON roles look similar to:

// app.json
{ "run_list" : ["role[rails_common]", "role[common]"] }

// logger.json
{ "run_list" : ["role[logger]"] }

// mysql.json
{ "run_list" : ["role[mysql]", "role[common]"] }

// web.json
{ "run_list" : ["role[web]", "role[common]"] }

Example of Ruby role file:

name "common"
description "Components required on every node"

run_list [
    "recipe[user::databag]",
    "recipe[deep_security]",
    "recipe[rsyslog]",
    "recipe[chef-solo-search]",
    "recipe[yum::epel]",
    "recipe[yum::remi]",
    "role[monitoring_client]",
    "recipe[git]"
]

  default_attributes user: { :data_bag_name => :users }

Scope

It is important to understand, that main target of this kind of setup is small to medium sort of applications -- tens of servers, rather than hundreds or thousands. But, at the same time, there are thousands of such applications, therefore ease of configuration and adaptability to the environment of next application is a key.

Project home

There is no code now in Github repo, as I write this, I only created new repo few minutes ago, but I am starting moving bits and pieces from other places. Watch this space.

Speeding up Jekyll site

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 29 Oct 2013 00:00:00 +0900

Recently I've added some optimizations to my Jekyll setup for faster page loading. Google and other search engines consider page load speed when building their ratings. It means that improving load speed can boost your SEO rating too, not only your users happiness.

Understanding the problem

Most of the optimizations in my case was about shrinking CSS, Javascript and images, removing blank spaces from HTML and reducing number of HTTP request by joining assets into single asset file.

But before doing optimization it is better to understand where do you stand and what do you want to optimize.

Good page speed analyzer can be help with the task, I've found decent one at http://gtmetrix.com/. There are others as well, for example https://www.site24x7.com/web-page-analyzer.html, Google has some too.

GTMetrics analyzer is pretty good, it grades the page by relative standing of your page to average page stats on the net, and can grade importance of each metric and recommendations for speed improvements. is Whatever comes red on their report is probably worth your attention.

In my case the most critical things where decreasing number of HTTP request for CSS and Javascript files and reducing size of the hosted images.

Assets pipeline

Javascript and CSS handling

I was quite pleased to find for the assets handling jekyll-assets gem that works with Jekyll out of the box, and it handles assets very similar to the way Rails does it, so less learning for me. There are other solutions as well, obviously I haven't tested all of them, but among tested ones this one is on of the better ones.

Adding the gem to Jekyll setup is quite simple, just include lines into Gemfile and _plugins/ext.rb accordingly:

gem 'jekyll-assets'
    ...
require 'jekyll-assets'

Next step is a bit of reorganizing of how Javascript and CSS (or SASS) files are laid out in your working directory.

In my setup JS and SASS files are in ./js/ and ./sass/ directories. Sass saves generated CSS's in ./css. For generating CSS files during local testing and when publishing I had sass command in my Rake task — in more details this was described in previous post — but with introducing assets pipeline this became redundant.

Examples of the new Rake tasks are given below.

Refactoring assets

Jekyll-assets introduces new directory _assets with subdirectories stylesheets and javascripts. I've moved all my .js and .sass files to the new location and replaced all style and script references from the code with just 2 lines:

{% javascript app %}
{% stylesheet app %}

Jekyll-assets uses similar to Rails assets declaration, so to include all of my JS and CSS, what I had to do is to create two new files — one in each directory.

File _assets/stylesheets/app.css

  #= require cssnormalize-min
  #= require dmytro
  #= require pygments
  #= require fancybox

File _assets/javascripts/app.js

  #= require jquery
  #= require jquery.tagcloud
  #= require fancybox
  #= require tag_cloud

Resulting parsed JS and CSS tags look like:

<link rel="stylesheet" href="/assets/app-e4c2b9c81e41368c0fcc779280bd5530.css">
<script src="/assets/app-b80c4942bb62b4e63380a8693c39923c.js"></script>

Optimizing images

Another important aspect of optimization is reducing file size of the images. There are two parts in it: reducing geometry of the (mainly) photos, and second is adding some optimization to reduce file of the image. Both JPEG's and PNG's can be reduced quite significantly.

Reduce geometry of JPEG files

First of all, files exported from iPhoto or such, can end up in a lot of various sizes. Sometimes you just forget to select proper reduction and have image files of impractical for the web sizes. ImageMagick can handle this quite easy, only needed is to add simple Rake task for it.

    def resize name, w, h
        sh %{mogrify -resize #{MAX_GEOMETRY[:w]}x#{MAX_GEOMETRY[:h]} '#{name}'}
    end
    
    MAX_GEOMETRY = { w: 1280.0, h: 960.0 }
    namespace :jpeg do
    
        task :list do 
          @jpgs = Dir.glob("**/*.jpg") - Dir.glob("_site/**/*.jpg")
        end
    
        desc "Scale down images to max geometry allowed"
        task :resize => :list do 
          files= { }
    
          @jpgs.each do |jpg|
            IO.popen "identify '#{jpg}'" do |p|
              f = p.read
              name,bla,size = f.split(/ /)
              name.sub!(/jpg\[\d+\]$/, 'jpg')
              w,h = size.split('x').map(&:to_i)
              next unless name && w && h
              resize(name, w, h) if w > MAX_GEOMETRY[:w] || h > MAX_GEOMETRY[:h]
            end
          end
        end

Optimize JPEG's and PNG's

Additionally to geometry reduction significant size reduction can be gained by optimizing imgages: removing unnecessary Exif tags and comments from JPEG's or applying various deflating algorithms. For PNG's optimizations are lossless, but for JPEG's you can select to reduce image quality or do lossless optimization.

Binaries for image optimizations are available for both MacOSX and Linux. On MacOSX jpegoptim is available as part of Homebrew and is installed simply by brew install jpegoptim. As for the PNG optimization pngout binary is downladable from Ken Silverman's Utilities page.

desc "Compress JPEG images"
    task :minimize => :list do 
      @jpgs.each do |f|
        sh "jpegoptim --strip-all --totals -o '#{f}'"
      end
end

namespace :png do
[... taks list omitted ...]

desc "Compress PNG images"
task :minimize => :list do 
  @pngs.each do |png|
    before = File.size png
    sh "./bin/pngout #{png} -q | true"
    puts "#{png} : #{before - File.size(png)} bytes smaller "
  end

HTML compression

For further reduction of the web page(s) size stripping all excessive white spaces and html comments can help too. There's a Jekyll plug-in (more than one) for that too. I've tried couple of them and stopped on jekyll-minify-html.

It is also installable as gem. To start using in, just add require 'jekyll-minify-html' statement in _plugins/ext.rb and env: production in _config.yml. That's it.

However I find it quite heave and usually comment out when previewing page locally. That's brings another topic of Jekyll slowness, see below.

Rake tasks

Finally, these are new Rake tasks for publishing site to Github and for building it locally for preview.

Rake run

This task starts locally Jekyll for previewing site on the local machine.

desc "compile and run the site locally"
task :run do
  pids = [ spawn("jekyll serve --watch --drafts") ]
 
  trap "INT" do
    Process.kill "INT", *pids
    exit 1
  end
 
  loop { sleep 1 }
end

Rake publish

Build site and publish it to Github. In this task I also use jpegoptim to optimize all thumb-nails from Jekyll GalleryTag plugin which are generated automatically on page build (on line 6).

desc "compile and publish the site to Github"
task :publish do
  sh "git checkout source"
  comment = %x{ git log -n 1 --no-merges --format=%s%b }.chomp.strip
  sh "jekyll build"
  sh "jpegoptim --strip-all --totals -o _site/images/galleries/*-thumb*"
  sh "cd _site && git add -A && git commit -m \"Publishing at $(date): #{comment}\" && git push origin master"
end

Some stats ...

Just some stats about size reduction as result of these experiments. There numbers are far from being precise. During all these changes I kept adding posts and images to the site, so it's hard to tell exactly how much space each one have saved. I only can roll back my git repo and see how size changed between commits.

	Before	After
Full page generated size (kB's)	483,524	478,324
Number of image files	375	878
Assets size (kB's)	88(css) + 184(js)	188 (JS+CSS together, including GZ'ipped copies)
GTmetrix Y Slow grade	C	B

Even with number of photos (main size factor on this site) increase about twice overall size went a bit down, and as for JS and CSS total size reduced by about 40%, not mentioning that instead of about dozen HTTP requests there are only 2 now.

... and problems

Jekyll is slow

After I started playing with multiple Jekyll plugins, I've started noticing significant slowing of the page generation. Even for small page changes it takes multiple tens of seconds to recreate file and page reload often show missing file or not regenerated CSS's.

Full page build with CSS/JSS/HTML compression tames more than 2 minutes, and this is only for 48 blog posts (this one is 49's).

Liquid is quite limiting

Another problem is that Liquid templating used by Jekyll is, while easy to learn one, is quite limited and a bit hard to extend.

I am finding that doing even simple (not-supported) things in Liquid is virtually impossible without writing custom plugin. Things, that in ERB would be done with 2-3 ruby commands, requires a lot of jumping through the loops.

Middleman?

Time to look for something new?

Recently I've read some intros to Middleman.

So far my impressions are:

ERB vs Liquid is GOOD THING. I can compare this relationship to that one of Puppet vs Chef, i.e. custom configuration language plus some added on top of it Ruby DSL vs clean from the start DSL;
things, that require a lot of plumbing in Jekyll are by default in Middleman (one example, being Asset Pipeline).

Oh well...

More experience with Jekyll and setup changes

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 13 Aug 2013 00:00:00 +0900

New features added to the Jekyll gem, at the same time trying to extend my Jekyll setup I hit Github limitations. Several times I've tried to introduce plugins to Jekyll setup or some advanced feature, only to discover that Github disables some of the (quite useful) features. .

Github's Jekyll

Plugins

Use of plugins is disabled by Github for security reasons. End of story. You can't use Jekyll plugins when relying on generating your site by Github.

Jekyll v. 1.x

First of all, there's a lot of changes in Jekyll. I was running v.0.12.x and many changes happened since Jekyll moved to v.1.x, one important change is added option to keep some files from overwriting.

Below is step by step process for publishing site with Jekyll running locally.

Process

You'd need some changes to the configuration of Jekyll.

Files

Configuration for included or excluded files for Jekyll generation:

File lists in Jekyll changed from simple list of space separated strings to arrays of strings. So, for example exclude list now looks differently (see below).
Add keep_files your _config.yml file. It will preserve your _site/.git repository information. This is, BTW default setting.
Make sure to include lsi attribute, so that your local Jekyll will use it for related content generation.

keep_files: [.git]
exclude: [bin, spec, Rakefile, Guardfile, _posts/template.md.erb, Gemfile*, \
  template.md.erb, sass, presentations/css/theme/template]
lsi: true

Setup git branches

Publications pushed to Github pages depend on the type of Github page. Here I am discussing "user pages".

User pages use master branch of the git repository. If master branch contains Markdown, Textile or other markup files Github's Jekyll will try to generate _site from it, but it won't try to generate anything if it's only HTML files.

We'd need to prepare branches accordingly:

generated content should go into master branch;
all sources go into different branch. I've chosen to call it source.

#
# Create source branch from current master
#
git checkout -b source 
#
# And push it to source branch at Github
#
git push origin source -u
#
# Now, *before* pushing to remote master, clone it into _site directory.
#
git clone -b master git@github.com:dmytro/dmytro.github.com.git _site
#

Add publishing task

Include this code into your Rakefile. You'd need to change/add some shell commands accordingly.

task :publish do

    # Just make sure we are generating from source branch
    sh "git checkout source"
    
    # I am using SASS for my CSS generation. You'd probably need to change this.
    #
    sh "scss sass/dmytro.sass:css/dmytro.css sass/style.sass:css/style.css"
    
    #
    sh "jekyll build"
    
    # 
    sh "cd _site && git add -A && git commit -m \"Publishing at $(date)\" && git push origin master"
    #
end

Build new hosts with Capistrano and Chef

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 21 May 2013 00:00:00 +0900

Include Chef recipes into Capistrano deploy script, and create full-blown application servers from scratch in a minutes .

I was using Chef-solo to configure new hosts before they can be used to deploy applications by Capistrano. But Capistrano is not just for the app deployment! .

Call Chef from Capistrano

Capistrano is capable of executing batches of commands in parallel on multiple remote servers, and its only dependency is SSH access. So, why not use this to spawn chef-solo processes on these servers?

So, easy solution is to simply include your Chef-solo repository/scripts as a first step in the Capistrano deployment process. This way each time you do a deployment Chef will take care about configuring your server first. And if your server is not configured, or even just recently built, all necessary components will be magically deployed with simply typing cap deploy.

Capistrano Recipe

Add this simple recipe to capistrano to make sure that Chef solo is started before any other Capistrano tasks:

set_default :chef_solo_path, File.expand_path("../chef-solo/", File.dirname(__FILE__))
set_default :chef_solo_json, "empty.json"

namespace :chefsolo do 
  task :deploy do
    temp = %x{ mktemp /tmp/captemp-tar.XXXX }.chomp

    run_locally "cd #{chef_solo_path} && tar cfz #{temp} . "
    upload( temp, temp, :via => :scp)
    run_locally "rm -f #{temp}"

    run "mkdir -p ~/chef && cd ~/chef && tar xfz #{temp} && rm -f #{temp}"    
    run "#{try_sudo} cd ~/chef && bash ./install.sh #{chef_solo_json}"
  end
  before "deploy", "chefsolo:deploy"
end

Just make sure that your chef-solo repository is installed in the ./deploy/chef-solo directory or set :chef_solo_path attribute accordingly.

Full text of Chef-solo Capistrano recipe is available in the repository in chef_solo.rb file.

Chef-solo Repository

This is Chef-solo repository that I am using for my deployments. I forked it from somebody else, but heavily modified to suit my needs. Currently install script supports deployment/bootstrapping to MacOSX and several Linux distros. Among supported Linuxes are Debian/Ubuntu and RedHat/CentOS. Wan not tested with Fedora (maybe needs some adjustments for the Rpmforge).

Server Bootsrapping

Even if you do not need to install any additional software using Chef-solo, you still need to put some prerequisits on the server. Like Ruby for Rails development - is an abvious example.

Chef-solo install script can do just that. Without any cookbooks, it would simply create development or production environment by installing: prerequisite development environment (development RPM's or DEB's), Ruby version manager - RVM, Ruby itself and Chef (in that order).

For this simply set :chef_solo_json attribute to "empty.json" as in the example above. "empty.json" is exactly what it says it's an JSON file that does not contain any recipes.

More about custom CSS for RT4

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 15 May 2013 00:00:00 +0900

How do I add custom CSS to RT, where is it? . I've received a question to my previous blog - RT can be less ugly. Question basically was, OK I have CSS, now where do I add it? .

Yes, it's not that easy to find this in RT interface. So, here is a small screen shot that shows how to get to this setting. Just don't forget you need to be logged in as root or as user who has permission to see this configuration menu.

Select menu: Tools » Configuration » Tools (again) » Theme

When you're there, you'll find Custom CSS (Advanced) text box, where you need to add your custom stuff:

From the same place you can customize some other visuals of the RT, like colors and company logo.

Here's scren shot for colors and fonts GUI confiruation:

Ruby 2 test drive

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 18 Mar 2013 00:00:00 +0900

Ruby 2 is fast! . First experience of running some of my scripts in Ruby 2 was: WOW !!! .

Ruby 2.0.0-p0 is barely out. I was wating for it to appear on RVM radar and today tried it finally. And ... as I said WOW !!!

Look at the numbers below. It's not what you can call a scientific approach for benchmarking, but demonstrates main idea well enough.

Running same small script in 2 Ruby revisions. Ususally second run is faster since some libraries are already preloaded in memory.

Can you see the difference?

ruby-1.9.3-p374

  Run          First       Second 
               ------      -----
  real         2.12        1.68
  user         1.54        1.53
  sys          0.16        0.13

ruby-2.0.0-p0

  Run          First       Second 
               ------      -----
  real         0.89        0.66
  user         0.59        0.56
  sys          0.10        0.09

Nagira v0.2.5 release

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 15 Mar 2013 00:00:00 +0900

New release of Nagira RESTful API includes number of important features. Nagira version 0.2.5 is pretty big release, including number of significant changes.

So I decided to jump version a little from 0.2.1 to 0.2.5

And here is all the new wonderful features included in the release:

Background parser

Which speeds up all HTTP requests by parsing of the Nagios status file in separate thread.

If you have a very big Nagios environment, parsing of status.dat file can take up to few seconds. Even though Nagira had a timed-parse feature before -- Nagira kept parsed data in a cache, and was not attempting to parse status more often than once in N seconds, once in a while unlicky user would have to wait for status update for several seconds.

This is eliminated now by parser running in the background and doing its job in separate thread. Meaning that answer to all user queries are fast. This, of course, can put some extra load on the server, since it will need to do the parsing regardless of whether ther are any requests coming, so you need to tune your configuration depending on Nagios environment.

Partial ActiveSupport

Meaning that you can use Nagira now directly from Ruby on Rails application.

ActiveSupport is indeed partial, more to come. Only pluralized nouns of the resources are supported now. What that means is that you can have your model defined like the following and get all your services information by Service.all:

class Service < ActiveResource::Base
  self.site = "http://localhost:4567/_status/"
  self.element_name = "service"
end

Only #all is supported at the moment (or #find(:all), which is the same).

Flexible configuration

More run time configuration options and environment varaibles available.

It easier now to control how Nagira is run using shell environment variables instead of going to source code. This makes it easier to use for non Ruby users, as well as simplifies some of the start-up scripts and configurations, since now they are plain files and not ERB templates as it was in the previos releases.

Additionally more configuration options are available in Nagira with thid release.

For the full description of configuration see Nagira documentation and comments in Nagira defaults file. But here's list of configuration options available: port, bind address, UNIX user to run Nagira, RVM setting, TTL for data, enable or disable background parser, log file and Nagios main configuration file locations.

Nagira is gem now

Finally the most important feature of this release is that Nagira is now a Ruby gem. It is available for the standard RubyGems location. It can be downloaded from the site or installed using standard gem way or included into Gemfile of the Ruby or Ruby on Rails project:

gem install nagira

Nagira relies on ruby-nagios library for parsing Nagios files. There was a big chunk of custom development, that I did for the ruby-nagios. Finally all custom development submitted to upstream project, merged in and I can use ruby-nagios gem instead of relying on my custom git branch. This made easier to refactor Nagira and convert it into gem.

It's UNIX my dear Watson

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 13 Mar 2013 00:00:00 +0900

Summary: Shell script for detecting aspects of UNIX OS: version, Linux distribution, clone name, release .

In many shell scripts one can find pieces of code such as:

if [ grep -i centos /etc/redhat-release ]
then
   DISTRO=CentOS
else
   ...

and so on.

It's not difficult task to find what OS/ditsro you are running on, but sometimes it takes couple minutes to remember what is actually different between Debian and Ubuntu to make reliable guess. And it is a repeating task, meaning that couple minutes can be spent more than once.

Meet Sherlock OS

This is a simple shell script to detect UNIX/Linux OS and various aspects of the OS. Especially for Linux: distribution type and derivative (such as CentOS/RHEL or Debian/Ubuntu)

But it is packaged as real Ruby gem (I am mostly developing in Ruby these days), so installation is as simple as gem install sherlock_os. After you install it you can do it like:

$ sherlock
OS=Linux
MACH=x86_64
KERNEL=2.6.32-5-amd64
DISTRIBUTION=debian
FAMILY=debian
DERIVATIVE=Debian
RELEASE=6.0.6
CODENAME=squeeze

or even

$ eval `sherlock | grep DERIVATIVE`
$
$ echo DERIVATIVE
Debian

Documentation

Available at http://dmytro.github.com/sherlock_os

Source code

http://github.com/dmytro/sherlock_os

Enjoy!

Memorizable password generation

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 08 Jan 2013 00:00:00 +0900

I was in need to write function to generate pseudo-random strings that are easier to remember and type for a human. To use either for passwords or let's say random coupon codes.

In my search I came across this solution. It sure works, but does not look pretty enough.

def self.random_password
  c = %w( b c d f g h j k l m n p qu r s t v w x z ) +
      %w( ch cr fr nd ng nk nt ph pr rd sh sl sp st th tr )
  v = %w( a e i o u y )
  f, r = true, ''
  6.times do
    r << ( f ? c[ rand * c.size ] : v[ rand * v.size ] )
    f = !f
  end
  2.times do
    r << ( rand( 9 ) + 1 ).to_s
  end
  r
end

So I decided to do my own. After few modifications I came up with example below:

class Array
  def rand
    self[Random.rand(self.size)]
  end
end

def random_password size=4
    c = %w(b c d f g h j k l m n p qu r s t v w x z ch cr fr nd ng nk nt ph pr rd sh sl sp st th tr) 
    v = %w(a e i o u y) 
    (0...size).reduce('') { |r| r << c.rand << v.rand }
end

All together it's not really any shorter, but function itself is less than half the size of the original, and as a side effect, there's also Array.rand method.

New tricks and questions with Jekyll

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Sat, 22 Dec 2012 00:00:00 +0900

Some new feature added to the blog, but this also brought new questions . I've learned something new about using Jekyll, but at the same time have some new questions not resolved yet .

Tagging and tag cloud

I was playing a bit with different versions of tags clouds. Some of them didn't quite work for me , since they require adding Jekyll plug-ins and Github does not allow running custom plug-ins.

I've ended up with jQuery based tags. Now there's a tag cloud on pages, cross referenced links to tags list. And question:

How to make tags in pages work?

For now site.tags lists only tags in blog posts, but not tags defined in other pages (for example in projects pages). however each individual page list tags when accessed by page.tags.

Ideally I want to see list of all my pages in the tags list. Not found solution for this yet. Keep looking.

Sources

All tag cloud related stuff can be downloaded from the Github repo and is in _includes/tag_* files.

Syntax highlight with Pygments

Up until today I was using Gists for inclusion (colored) pieces of code into blog posts. This works quite well, but for many small snippets can be a bit troublesome. Instead I've discovered for myself Pygments, and this is the result:

1 def foo
2   puts 'foo'
3 end

To have that code above colored, simply add highlight tags around your code as in the example below:

{ % highlight ruby linenos % }
  # ....  code here
{ % endhighlight % }

You need to do some preparation too:

add pygments: true in _config.yml
create CSS file and include reference to it
- CSS file can be generated easily by: pygmentize -S default -f html > default.css

Nagios Checks Aggregation

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 14 Dec 2012 00:00:00 +0900

Summary: Reviewed several patterns for integrated monitoring with Nagios and Nagira . Nagira — RESTful API for Nagios — simplifies Nagios checks aggregation . For second part of the series an easy way of combining several Nagios results into aggregated check is presented.

Nagios Checks Aggregation

This kind of setup is applicable in situations where there is a need to integrate several already existing Nagios checks, but you need to aggregate information from checks into combined metric. In this case all necessary information already exists in Nagios, there is no need to read it from monitored devices, API can fetch source information from Nagios status store and write back calculated output. See diagram below.

Computer Rack Prower Monitoring

An example of such setup is an electric current monitoring system.

Electric current of the power lines in a computer rack are monitored by Nagios. Information from power lines is collected by SNMP. Computer racks have two power inputs for redundancy; servers, having similarly two redundant power supplies each, are connected to both power inputs.

However, additional requirement exist for monitoring gross current consumed by each rack. For example, in the case of one side power source failure, the whole rack power can fail if sum total of currents is higher than power limit for the remaining source — even if each of the power lines before the failure is in a green zone.

In this case, to obtain data for the pair of power sources, data of each individual line is read from Nagios status database, instead of using SNMP. Data processed by script, and if total gross current of two power lines is greater than current limit for single line, status is marked as WARNING or CRITICAL. Resulting combined metric (sum of two currents) is written back to Nagios together with RAG status. All communication — reads and writes — are done only using web-services Nagira API.

Possible uses

Other examples where such kind of setup can be applied are:

load balanced or DNS round-robin farm of web servers, to monitor total network traffic, not only traffic on individual servers;
status of clustered application — if it is known, for example, that application execution requires X number of servers with Y CPU cores, each occupied by Z%.

See more

Monitoring Applications with RESTful API

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 12 Dec 2012 00:00:00 +0900

Summary: Reviewed several patterns for integrated monitoring with Nagios and Nagira . Nagira — RESTful API for Nagios simplifies integration of distributed Nagios systems . This is first installation of the series.

Nagios is great system. Many features in Nagios are superior to many open source and commercial monitoring system. Adding HTTP based API for Nagios opens new possibilities for simpler architectures of distributed monitoring with Nagios.

Nagira API provides access to system status, objects configuration in Nagios as well as provides an interface for submitting passive check results using the same HTTP protocol. This opens wider capabilities for monitoring system architecture modularity and integration.

Applications with HTTP RESTful API

Here I want to review one of the simplest patterns with Nagira - monitoring applications that have built-in HTTP API (RESTful API in our case). This pattern was implemented recently in new project Nagios For Couchbase. Similar approach can be adopted for any kind of application, that provides access to application metrics via HTTP API.

Nagios For Couchbase

Couchbase is advanced memory based NoSQL key-value database. It has its own management RESTful API and API for data access. The API provides access to most if not all of the Couchbase operational metrics. Couchbase also has nice web console with multiple graphs displaying server metrics in real time. Actually web user interface in Couchbase is implemented as jQuery interface built directly on top of JSON RESTful API.

Given Couchbase API combined with Nagira API, setting a monitoring system becomes a simple task. Architecture of such setup is presented on the diagram below.

System Architecture

Script connects to Couchbase API and retrieves server metrics as JSON data object (via HTTP GET request)
Script process data and
Sends data to Nagira API (HTTP PUT)
Nagira writes to Nagios external command interface
Results are then processed internally by Nagios

Advantages of this architecture as far as one can see are:

Since only HTTP protocol involved for both retrieving data and registering results, script that runs the checks, can be in any location — on Couchbase server, or on Nagios server or any other host on the network.
List of couchbase servers can be retrieved from Nagios configuration itself, if corresponding host-group is configured (for example 'hostgroup couchbase'). Configuration becomes completely data driven and requires no additional changes when adding/removing Couchbase servers to the pool.
JSON object containing multiple metrics can be read as with single HTTP GET request to Couchbase server, status for multiple checks can be updated similarly with single HTTP PUT to Nagira API, which produces multiple writes to Nagios command file. This reduces load on Nagios server, which is not dealing with scheduling and executing checks.

Executing Checks

How actually checks are performed will be reviewed in some of the following posts. Please come again!

Unobtrusive Git prompt for ZSH

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 27 Nov 2012 00:00:00 +0900

Summary

How to incorporate Git information in Zsh shell prompt in an unobtrusive manner

vcs_info module for ZSH gives an ability to design nice and functional prompts for use with version control systems. Google can find large number of references to manuals, tutorials, blog posts and what not regrading using the module for prompt designs. In many cases, however, as to my taste resulting prompts stand in the way more than actually help. Four or five lines shell prompts are not for me, same if prompt occupies 80% of the command line space, or it is too colorful.

It took me a while, first, to learn about ZSH prompts and vcsinfo, and second to design a prompt, that I can use without too much hating it. Below are some screen-shots and code for the prompt.

I am using git mostly, so all my examples below are for git.

Screen shots

Standard prompt

When not in the git repository prompt is pretty much standard UNIX shell prompt: user name, hostname and last part of the current directory. Stepping into git repository add right part to the prompt (see below).

Small screen

Same prompt if I am in the git repository but my screen space is too small to accommodate both parts of the prompt (yellow line on the right is pane border in Tmux). Also what is really nice about right part of the prompt: if I start typing very long command and come close to the right part of the prompt, it disappears.

Clean Git repository

When I am inside git repository right part of the prompt appears and it indicates:

git branch
git repository name
directory inside the git repository (last part of it)

Git repository with uncommitted changes

Right part in this case adds indication for:

U - for unstaged changes (encircled)
S - for staged changes
M - for stashed changes

Action indication

Finally last screen shot shows example of unfinished merge. In red color indicated current action: merge, rebase,etc.

Source code

autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git
precmd() {
    vcs_info
}
 
setopt prompt_subst
zstyle ':vcs_info:git:*' check-for-changes true
zstyle ':vcs_info:*'    formats "%f[%%n@%%m %1~] $ " "%f%a %F{3}%m%u%c %f%b:%r/%S" 
zstyle ':vcs_info:*'    nvcsformats   "%f[%n@%m %1~]$ " ""
zstyle ':vcs_info:*'    actionformats '%F{5}(%f%s%F{5})%F{3}-%F{5}[%F{2}%b%F{3}|%F{1}%a%F{5}]%f '
 
PROMPT='${vcs_info_msg_0_}'
RPROMPT='${vcs_info_msg_1_}'

Short explanation for the code.

Each zstyle ':vcs_info:*' line sets style for vcs_info messages. There are 2 messages configured: vcs_info_msg_0_ and vcs_info_msg_1_.
In each line vcs_info_msg_0_ and vcs_info_msg_1_ are first and second arguments in each definition. I.e., for example on line 9 "%f[%%n@%%m %1~] $ " is msg 0, and "%f%a %F{3}%m%u%c %f%b:%r/%S" is msg 1.
- formats defines style for prompt in inside git repository
- nvcsformats - same for non git directories
- actionformats - for git actions
Left and right parts of prompt are set on lines 13 and 14 correspondingly to display vcs_info_msg_0_ on the left and vcs_info_msg_1_ on the right.

References

Below are links to some of the resources I've used. They can be used as reference or simply as example of different designs of the prompt.

Opening multiple Tmux panes in the same directory

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Thu, 22 Nov 2012 00:00:00 +0900

Summary

This small snippet of code allows opening windows/panes inside tmux in the same UNIX directory.

Source code

When working on project I quite often want to open new window or pane in Tmux in the same directory where I am now at the moment. Before I figured out how to do it, it was like this: open new pane, cd <...>, cd <...>, cd <...>. Even with single cd and tab completion, navigating through deeply nested project tree can be not that easy.

Solution

Here's the small shell function and alias to help.

This is how I use it

When I want new window or pane to be created and cd to that dir automatically I simply type pwd and after this open new window/pane.

And this is how it works

tmux_pwd function executed in your current directory simply sets option for window in your current Tmux session to use this directory as default (lines 3 and 4 of the gist).
- Line 3 detects current session of the tmux.
I don't want this function to be executed in the shell which does not run under Tmux. Hence the line No 1 in gist.
And I don't want all my new windows to be opened in this directory forever. I have my 'global' default directory, which is ~/Development, and this is where all my windows usually open. So, there's a 'timeout' setting to set my default directory back after 5 mins. Lines 5 and 6.
And finally there's an alias, on line 8.

I've added this to my zsh functions, but I don't see why it shouldn't work in another shells like Bash.

My Really Low Tech Music Player

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 20 Nov 2012 00:00:00 +0900

Summary

Small script to play music collection from commands line.

Really minimalistic Ruby script to shuffle audio files and play them randomly form disk No GUI, very little contol, just enough for simply playing music. Script randomizes list of files and plays them in random order, Ctrl-C skips current song and goes to the next, Ctrl-Z exits script.

Source code

Configuration

Before using script generate list of songs with:

find PATH/TO/THE/LIBARY -type f > music.list

Creating "playlists" is just grep:

grep Beatles music.list > favorites.list
grep "Deep Purple" music.list >> favorites.list

Start multiple synchronized SSH connections with Tmux

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 31 Oct 2012 00:00:00 +0900

Summary

Small shell script to start multiple connections to SSH servers with synchronized keyboard.

If you need to connect to multiple SSH machines and perform some operation on all of them, there are many ways to do this. Typically regular sysadmin would write simple for one-liner in shell:

for i in $HOSTS; do <....> done

It's OK for simple command or couple of commands. But, if commands produce verbose output, or you want to see what's happening in between your commands, this approach can be more troublesome. Or, for example, your ssh keys are not set and you'd need to type password for each SSH prompt.

Tmux way

Using tmux you can do it in a different way. If you haven't started with tmux yet, there's a good reason to do it.

This script allows with single command open multiple equally split-panes in Tmux window and synchronizes keyboard input in all windows. Anything you type will go at the same time to all tmux panes and you will see output of commands in all windows as they are executed. Any time it is you can break keyboard sync between panes if you need to execute something in single shell only, then you can sync again and continue as before.

Full text of the script is provided in the Gist below. It is based on solution I've found online, a bit simplified and extended to accept list of hosts from command line.

I usually run it as:

ssh-multi myhost1 myhost2 myhost3 myhost4

or for some groups of hosts, I put list into plain text file:

ssh-multi $(cat list-of-hosts)

The script is included in dotfiles repository. You can eiterh copy-paste from included Gist, or simply get the whole thing, by cloning o downloading it using links below.

Script source code

What's in the name -- Nagira?

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Thu, 11 Oct 2012 00:00:00 +0900

When a while ago I was selecting name for my new project I didn’t have slightest idea that Nagira is actually somebody’s name.

Nagira sounded to me as a name, I thought it was more like a female name, maybe of Persian origins. It was a bit of surprize for me to type ‘nagira’ in google and see more than 100 000 hits.

So, I’ve discovered that ‘Nagira’ is a name, but not female, and not Persian, but rather Japanese and male.

So, today on my way to work I see this real life example – Nagira’s concert in Asakusa Jinja.

Below you can see references to the project of mine: Nagira — Nagios RESTful API.

My first gem and first Rubygems impressions

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Sun, 09 Sep 2012 00:00:00 +0900

Two days ago I've produced and published my first gem.

Process ? Simple!

Being from the Perl CPAN background I was prepared for something more laborious: tar'ing and gzip'ing, and uploading files to web server. Moving file(s) around, maintaining archives and deleting old versions (not that I've got any old versions yet).

However, in the end it was much simpler. Even little disappointingly too simple. After you finished writing gemspec file, you simply do gem build [gemfile] to build it, gem install ... to install and test locally and gem push [just-built.gem] to push it to <http:://rubygems.org> . You'd need to create your Rubygems account first.

After last action -- having answered command line prompt questions about your Rubygems login -- few seconds later you think: "That's all!?" Yes, that's it. Your gem released and can be installed simply by gem install [name].

Well done - Rubygems

After registering and playing a little bit more with Rubygems, I've found some nice features.

Documentation

Rubygems displays YARD-formatted documentation for the gem. I am not sure if it's by default or because I had rake doc task in my Rakefile, that call yarddoc.

Stats

Looking 2 days after uploading gem I discovered this download stats. Of course, you'd like to know, whether somebody likes your work or even did at least somebody download at all ?

What about CPAN?

What I found missing in Rubygems compared to Perl CPAN, or at least haven't found it yet with my limited Rubygems experience.

Testing After each update of CPAN module I've used to receive automated test results. Module automatically submitted to testing framework and your test are being run in multiple exnvironments. Good for testing on platforms that I don't have access too -- Windows or something like AIX/HP-UX, or testing for missing dependencies -- did I have that_nice.gem installed on my system, but didn't include it in dependency list?. However, latter case is rather less important with bundler and Gemfile.
Namespacing CPAN allows you to apply and register namespace for the module(s) and families of modules. By setting some rules on namings Perl does it easier to find what you really need going from top to bottom. This is what I really miss in Ruby community - searching for right gems can be simpler then it is now.
Ticketing RT (AKA Request Tracker) is ticketing system developed by Perl programmers and it is used by Perl programmers. What can you say more? It is ticketing system used by all Perl contributors. Will be nice to have something similar for Ruby and Rubygems.

What I have released?

Hopefully something useful. It's an RSpec test suite for what I call Normalized Hash. Description of Normalized Hash and more info about the specs are in github project.

RT4 can be less ugly

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 18 Jul 2012 00:00:00 +0900

Default ticket display in RT is really ugly – all system messages (actions) are shown exactly the same way as user actions. Luckily RT4 has added possibility to have custom user CSS. This addition to CSS make system actions to stand out less: smaller fonts, and less contrast.

Stuff that’s added is all .ticket-transaction entries at the bottom.

Now I've got my environment ...

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 18 Jul 2012 00:00:00 +0900

It looks like I finally have an environment I am satisfied with and I have tools to manage it.

Long previous years I had set of my own configuration files that I carried with me from one environment (usually associated with new job) to another. These were huge .emacs file, even bigger .fvwm2rc file, lot of additional e-Lisp files, .bashrc and so on.

Management system was simple: got a job → send yourself email from home with lot of attachments → save to home directory → add changes. Usually after ‘add changes’ step, one had to copy changes back to home files.

Now, I’ve got my own copy of dotfiles on Github and that makes life enjoyable again. There’s README file that explains most what you need to know to start using it.

At about the same time I’ve discovered several new things – iTerm2 and tmux.

iTerm2 + tmux + emacs

Making navigation in all three of the above if not the same then at least similar makes life much easier. Just to make things clearer: I am not a mouse person, and by navigation I mean using keyboard.

Since I am mostly in the Mac now, I am using FWVM much less, and navigating in it is less concern for me.

Additionally navigation includes more things then jst switching from window to window. There’s navigating on command line: search command history, search and scroll forth/back in scrollback buffer, etc.

Navigating in iTerm2 and tmux

Most of the keys set by default in iTerm2 (Cmd-, Cmd-End?Home etc.) are unused when iTerm2 used in combination with tmux. For example, when using tmux scroll buffer, iTerm2 scoll buffer cannot be used and only messes things up. So, I actually set iTerm2 buffer size to 0 — this removes iTerm2 scroill bar as well, and this also gives a bit more screen space.

Similarly with all other features. To say the truth I actually use only 2 features of iTerm2: full-scrrn mode and ability to map keys to send text strings or sequences of hex codes.

I’ve deleted all default key mappings from iTerm2 and remapped them to send tmux commands instead. There are 2 places you have to do this:

Profiles keymaps

By default iTerm2 loads xterm default key mappings. They intervene with global key mappings and override them. The simplest solution I’ve found to be simply delete all of them. I haven’t run into any problems as of yet related to this, YMMV.

Go to iTerm2 preferences

Default (global) keymaps

Same with the global key mappings — I use none of them in tmux — I deleted all defaults and set my own.

However, here instead of doing all manually, on all macs, I can reuse my config by saving presets to file, and by re-imporing it into iTerm2 preferences. I can also use github and dotfile repository to track changes and share presets across all environments.

I’ve added 2 Rake tasks for exporting, importing iTerm2 global key presets:

    rake iterm:keys:export  # Save global key mapping to file
    rake iterm:keys:import  # Load global key mapping from file to iTerm2 defaults

Actual command that is executed uses McOSX defaults command:

    defaults read ~/Library/Preferences/com.googlecode.iTerm2 GlobalKeyMap > iTerm2/GlobalKeyMap

Navigation key mappings

This is a list of key mappings configured. In the table below ‘screen’ refers to:

pane for tmux
window for emacs

Since emacs does not do distiction between Option and Command keys on Mac, ‘M-’ in emacs column refers to Meta key, which is Option or Command. For the same reason for screen jumping I had to use Ctrl- in Emacs.

Navigating windows

Function	`tmux`	`iTerm2`	`emacs`	Zsh
Go screen left	` Left	Cmd-Left	Ctrl-Left	–
Go screen right	` Right	Cmd-Right	Ctrl-Right	–
Go screen up	` Up	Cmd-Up	Ctrl-Up	–
Go screen down	` Down	Cmd-Down	Ctrl-Down	–
Go previous screen	` ;	Cmd-` and Cmd-;	Cmd-Up	–
Word forward	–	Option-Right	M-Right	Option-Right
Word backward	–	Option-Left	M-Left	Option-Left
Delete word forward	–	Cmd-d	M-d	Cmd-d
Delete word backward	–	Cdm-Delete	M-Delete	Cmd-Delete

Windows operations

Function	`tmux`	`iTerm2`
Go to window Number	` number	Cmd-“number”
Toggle panes sync	` S	Cmd-S
Split horizontally	` “-”	Cmd-“-”
Split vertically	` \	Cmd-“\”
Next window	` n	Cmd-Option-Right
Prev window	` p	Cmd-Option-Left
`Tmux` Help	` ?	Cmd-h

Tmux selection mode

Function	`tmux`	`iTerm2`
Mouse selection ON	`+m	Cmd-m
Mouse selection OFF	`+M	Cmd-M
Start `tmux` selection mode	`+[	Cmd-[
Copy `tmux` selection to system clipboard	`+Ctrl-]	Ctrl-Cmd-]
Paste `tmux` selection	`+]	Cmd-]
Start selection mode, scroll page up	`+PgUp	(TODO)

Various tmux actions

Function	`tmux`
Prefix	`
Change prefix to Ctrl-o	F12
Change prefix to “`”	F11

Note: It is good to have option of disabling “`” as tmux prefix key for cut-and-paste shell scripts into terminal. Most of the mappinngs will not work when prefix changed to Ctrl-o, since in many cases iTerm2 keystrokes send text “`”+something.

Can one little second kill your servers?

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 02 Jul 2012 00:00:00 +0900

Actually — yes.

Early Saturday morning Japan time most of the servers in the company where I work now started to act. CPU usage spiked to 100-200% while no load was actually hitting the servers.
Mysql on restart used again insane amount of CPU. At the same time, servers were quite responsive and acting like normal.

After some time later reports started to appear in Google search… Reports were talking about … leap second

At the mid-night GMT time on Jun 30, 2012 extra second was added — leap second. NTP supposed to have been picked up that small change, but it did not. Most of the Linux servers’ CPUs went into timeless cycle: check flag (‘leap second’ flag in NTP protocol), set flag, check flag, reset flag …

The funniest thing about all this story is sequence of commands you’d have to run to fix the issue:

        ntp stop
        ntpdate ntp.pool.org 
        ntp start

Or even this (!!) :

        ntp stop
        date -s "`date`"
        ntp start

Making Debian 6.x play nicely with Cobbler

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Thu, 21 Jun 2012 00:00:00 +0900

Cobbler works OK when client system is RedHat-based — RHEL or CentOS — but support for other OS’s is less than perfect. Recently I had to configure Debian (currently at 6.0.5) to be built by Cobbler, and here is a HOWTO for making it work.

Problems

There are several issues when making Debian to be able to be installed by Cobbler.

First of all current (6.0.5) Debian netboot installer is broken. It fails with “No disks found” message, reason being that no IDE, SCSI or SATA udebs are included with netboot installer. This has been reported but not fixed yet. The only way to fix this issue seems to be rebuild the installer from scratch.

Additionally I wanted to avoid any questions asked by installer—make it fully unattended. This requires some addition to Debian’s netboot initrd file. Otherwise even if you provide seed file it keeps stopping and asks you about keyboard layout and network configuration. Reason is—these questions have to be asked before network is up and seed file from network can be loaded.

Process

Cobbler server

On your Cobbler server first of all import Debian as usual: cobbler import --breed=debian --name=Debian6.0.5 --path=...

Build server for the Installer

Build your Debian installer build server. Setup system using instruction at Debian Installer page

This involves installing dependencies, checking out Debian Installer and installing dependencies again (in some cases).

Install dependencies

Use Debian Installer Checkout page for the reference.

Installing installer basically performed with following commands:

 svn co svn://svn.debian.org/svn/d-i/trunk debian-installer
 cd debian-installer
 scripts/git-setup           
 mr -p checkout

Check dependencies are satisfied

 apt-get build-dep debian-installer
 dpkg-checkbuilddeps

In my case lat command complained about missing win32-loader >0.7.2. This is package from Wheeze (7.0) but Squeeze has only 0.6x. I had to download deb file from Wheezy mirror and install it manually:

 wget http://ftp.yz.yamagata-u.ac.jp/debian/pool/main/w/win32-loader/win32-loader_0.7.4.3_all.deb
 dpkg --install win32-loader_0.7.4.3_all.deb

Add/change configuration files

All configuration files reside below debian-installer/installer/build directory. All paths below refer to sub-directories starting from this point.

Kernel configuration

First is kernel, installer image must have same kernel version as build host, since kernel and modules are copied from build host to installer image. Make sure this is set correctly in ./config/<arch>.cfg. In my case, I am building 64-bit installer, therefore architecture config file is ./config/amd64.cfg:

Missing udebs

Next part is where actually broken installer fixed, here we will add missing drivers.

Netboot package lists are configured by files in pkg-lists/netboot directory. To make the change work for all architectures the easiest is to change pkg-lists/netboot/common file.

Below is list of additions to the file. Full listing of pkg-lists/netboot/common file can be found in Github gist

Pre-seeding

Automation of Cobbler installs is done by preseed file. Preseed file can resign on Cobbler server and configured by --kickstart option.

However when using netboot installer there are some configuration options that need to be set before kickstart is downloaded from Cobbler: console keyboard, language during installation etc. For this reason they need to be in initrd file.

There are several ways to do this. If you are using pre-built installer, you can add preseed.cfg file to it. Simple shell script can help with this.

But, since we need to make initrd ourselves anyway, it is easier to add preseed to the image during build process. Create preseed.cfg file with following content:

and add this line to config/local file:

    PRESEED = preseed.cfg

Build

Now everything is ready and we can build:

    make reallyclean
    fakeroot make build_netboot

Install

Copy all sub-directory dest/netboot/debian-installer/amd64 to Cobbler’s /var/www/cobbler/ks_mirror/Debian6.0.5/install/debian-installer/amd64 and sync:

   ( cd dest/netboot/debian-installer/amd64 && \
     tar cf - . | ssh root@cobbler  \
     "(cd /var/www/cobbler/ks_mirror/Debian6.0.5/install/debian-installer/amd64 && \
       rm -rf * && tar xf - && cobbler sync\
      )"                                      
   )

Done.

Як перетворити системний диск в case-sensitive

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Thu, 15 Dec 2005 00:00:00 +0900

З цієї сторінки

Create a case-sensitive HFS boot disk

‘’Authored by: jay2 on Tue, Feb 8 ’05 at 02:18PM’’

I’ve set up two macs (both running 10.3.x) with case-sensitive boot disks by doing the following:

use carbon copy cloner to make a bootable backup on a firewire disk
boot from the backup
open a terminal window and incant:
+ sudo diskutil eraseVolume “Case-sensitive HFS+” “Macintosh HD” /Volumes/Macintosh HD
+ (that’s all on one line)
restore from the backup to the newly formatted HD

cccloner makes it a piece of cake. i did this over a year ago, and have had no problems with case-sensitive hfs+.

Як воювати з Exim'ом

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 11 Apr 2005 00:00:00 +0900

Найкращий спосіб для настроювання поштових серверів давно відомий: стерти і встановити sendmail. Але якщо це з якихось причин неможливо — треба боротися з тим, що є.

Exim

Так само, як і postfix розрахований на те, щоб страшенно піклуватися про ресурси комп’ютера, на якому він працює. Навіть за рахунок того, що доставка пошти може зайняти кілька днів. Зате CPU працює і не пітніє.

Стандартна конфіґурація exim4 доставляє за раз не більше 10 листів. Симптоми: ви вмикаєте свого fetchmail’а після відпустки, в черзі кілька тисяч листів і вони починають поступати по 10 штук за кожні 10-20 хвилин. Розрахунок необхідного часу для отримання пошти залишається вправою для читача.

В журналі повно повідомлень подібних на таке:

2005-04-11 08:43:58 1DKm5u-0000Nq-U2 no immediate delivery: more than 10 messages received in one connection

Підказка : в Дебіані журнал exim4 знаходиться в /var/log/exim4/mainlog

Вихід: вставити в конфіґурацію exim’а рядок:

 smtp_accept_queue_per_connection = 0

Debian auto login

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Sat, 30 Oct 2004 00:00:00 +0900

Це повинно працювати на будь-якому з лінаксів. Випробувано на Дебіані (sarge).

Компілюється спеціальна програмка для автоматичного логіну


    # cat /usr/src/autodk.c 
    int main() {
               execlp( "login", "login", "-f", "dk", 0);
    }

потрібно змінити тільки ім’я користувача.

Компіляція

    # gcc -o autodk autodk.c
    #  cp autodk /home/dk/

Зміни до /etc/inittab

    # grep dk /etc/inittab 
    1:2345:respawn:/sbin/getty -n -l /home/dk/autodk 38400 tty1

Зміни до .bash_profile

Щоб автоматично стартував X там де потрібно (на першому VT) додати:


    # cat ~dk/.bash_profile 
    . . . 
    if [ -z "$DISPLAY" ] && [ $(tty) == /dev/tty1 ]; then
      startx
    fi
    
    #

Пояснення: запускає автологін програму на 1му вірт. терміналі.

Джерела

Інформація взята з ЛінаксГазет

Покращення швидкодії диска

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Tue, 12 Oct 2004 00:00:00 +0900

Це взяте повністю з форуму linux.org.ua

автор:
Псевдо: iip
Ім’я: iip
Розташування: Новачок
Дата реєстрації: 17. серпня 2004 17:24
Ел. адреса: схована

Чи є у когось якісь поради щодо покрашання швидкодії лінаксової системи?

Прошу поділіться тут.

Я нещодавно підвищив швидкодію свойого жорсткого диску за допомогою команди hdparam. Перш ніж приступити до удосконалення перевірте поточну швидкодію. Введіть у командний рядок з правами адміністратора (su – root, тощо) наступну команду /sbin/hdparm -tT /dev/hda

hda – це ваш перший жорсткий диск, hdb – другий і т.д.

Після того введіть /sbin/hdparm -c1 -d1 -m16 /dev/hda

Відтак знов /sbin/hdparm -tT /dev/hda і порівняйте показники.

Щоб установки застосовувались при кожному запуску системи вставте ваш hdparm рядок у файл /etc/init.d/boot.local.

Для перевірки режиму DMA, який використовується у вашій системі можна спочатку виконати hdparm -I /dev/hda, а тоді за допомогою модуля IDE DMA в Yast (в мене SUSE 9.1) примусити жорсткий диск використовувати правильний режим DMA (можливо udma2 або udma4 – програма скаже, якщо ваш привід не підтримує певного режиму).

Заголовок з відступом -- CSS

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 08 Oct 2004 00:00:00 +0900

На http://www.acooke.org/andrew/writing/email.html заголовки виглядають ось так:

CSS для цих загловків:

    div.indent {
      border: none;
      padding: 0.5em;
      margin-left: -5%;
      float: left;
      padding-right: 5%
    }

Нова мода podcast

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Wed, 06 Oct 2004 00:00:00 +0900

Ідея дуже проста — взяти радіопрограму, записати її в mp3 і засунути в iPod.

podcast відноситься до того, як це зробити автоматично. Тобто кожного ранку Ви змоєете на іПоді мати свіжий випуск новин з України, чи нічну програму Радіо Люкс:

як це зробити

VPNC на Маку

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 04 Oct 2004 00:00:00 +0900

Стандартний vpnc встановлений з MacPorts не підтримує кількох досить корисних конфігураційних параметрів. Ці параметри були додані Debian’ом, тому і підтримуються тільки самим Debian’ом та похідними від нього дистрибутивами.

Один з таких параметрів Target Networks дозволяє не додавати Default route до мережевого інтерфейсу vpnc – tun.

Цей скрипт дозволяє емулювати Target Networks в стандартному vpnc. Конфіґурація скрипта – пару рядків на початку файла.

Скрипт

Debian -- як зняти прапорець -nolisten

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 04 Oct 2004 00:00:00 +0900

Останній дебіан запускає Xserver з прапорцем -nolisten. Ось де собака зарита:



 $ ps …

   […] /usr/X11R6/bin/X […] -nolisten tcp […]

Це підвищує безпеку системи, але одночасно забороняє з’єднання з віддалених машин до сервера X. Зняти цей прапорець можна в файлі

 $ cat /etc/X11/xinit/xserverrc
 #!/bin/sh
 exec /usr/bin/X11/X -dpi 100 -nolisten tcp
 $

Якщо сервер стартує з XDM то потрібно відредагувати файл /etc/X11/xdm/Xservers. В ньому так само прибрати прапорець -nolisten:


 #:0 local /usr/X11R6/bin/X vt7 -dpi 100 -nolisten tcp :0 local /usr/X11R6/bin/X vt7 -dpi 100 tcp

Optical Illusions

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 04 Oct 2004 00:00:00 +0900

From /. 4 Oct 2004:

Science: The Goggles, They Do Nothing

Posted by michael on Sunday October 03, @07:24PM

from the do-not-view-while-operating-machinery dept.

Suchetha writes “In anticipation of a slow news day i would like to
direct the Slashdot hordes to
Akiyoshi Kitaoka’s Optical Illusions page. The page also has explanations on why/how they occur (in icky PDF format). The page is on a .jp uni server so they SHOULD be able to handle the herd of rhinos that is ./.”

Попередження з самої сторінки ілюзій — “якщо відчуваєте, що голова крутиться… тікайте з цієї сторінки”

Створення NFS експортів у MacOSX з командного рядка

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Mon, 04 Oct 2004 00:00:00 +0900

Створення NFS експорту, щоб його можна було змонтувати з правами rw для root’а

Як зробити це за допомогою командного рядка.

(1)

 # niutil -list / /exports
 81       /Volumes/Users

;зауваження: спочатку створюється фіктивна гілка з назвою servername, а потім замінюється на справжню точку експорту. Для того, щоб знати ID цієї гілки.

(2)

 # niutil -create / exports/servername
 # niutil -list / /exports
 81       /Volumes/Users
 134       servername

(3)

 # niutil -insertval / 134 name "/Volumes/NewExport" 0
 # niutil -list / /exports
 81       /Volumes/Users
 134       /Volumes/NewExport servername
 #

(4)

 # niutil -destroyval / 134 name "servername"
 # niutil -list / /exports
 81       /Volumes/Users
 134       /Volumes/NewExport
 #

(5)

 # niutil -createprop / 134 "clients" "myclient.domain"
 # niutil -createprop / 134 "opts" "maproot=0:0"
 # niutil -read / 134
 name: /Volumes/NewExport
 clients: myclient.domain
 opts: maproot=0:0
 #

(6)

 # ps ax | grep mount
  373  ??  Ss     0:00.28 mountd
 # kill -HUP 373

Джерела:

http://macosx.midata.fi/info-nfsexport/
http://deaddog.duch.udel.edu/~frey/darwin/archive/DarwinAndNFS.pdf

Помилка sendmail

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 24 Sep 2004 00:00:00 +0900

Наступні помилки в Солярисі викликані невірними дозволами на теку /etc.



        /etc/mail/sendmail.cf: line 77: fileclass: cannot open

        ‘/etc/mail/local-host-names’: Group writable directory

        /etc/mail/sendmail.cf: line 493: fileclass: cannot open

        ‘/etc/mail/trusted-users’: Group writable directory

Виправлення досить просте: chmod g-w /etc

20 years ago / 20 років тому...

dmytro.kovalov@gmail.com (Dmytro Kovalov) — Fri, 20 Feb 2004 00:00:00 +0900

The Macintosh uses an experimental pointing device called a ‘mouse.’ There is no evidence that people want to use these things.

John C. Dvorak, SF Examiner, Feb. 1984.

Dmytro Kovalov: Personal projects, blog, tips and tricks, UNIX development, administration and devops.

iPhoto and files permission

Is MacOSX 10.9 == MacOS 9.x?

Running Chef roles from Capistrano

Capistrano + Chef

Databag messenger

Hosts (AKA nodes) databag

Applying Chef roles

Capistrano task

Next steps

und Capiche

What is it?

Application Centric Architectures

Management

What is my goal for this project?

Chef and Capistrano

Cap + Chef = Capiche

Components

Current state

Scope

Project home

Speeding up Jekyll site

Understanding the problem

Assets pipeline

Javascript and CSS handling

Refactoring assets

Optimizing images

Reduce geometry of JPEG files

Optimize JPEG's and PNG's

HTML compression

Rake tasks

Rake run

Rake publish

Some stats ...

... and problems

Jekyll is slow

Liquid is quite limiting

Middleman?

More experience with Jekyll and setup changes

Github's Jekyll

Plugins

Related posts and LSI

Jekyll v. 1.x

Process

Files

Setup git branches

Add publishing task

Build new hosts with Capistrano and Chef

Call Chef from Capistrano

Capistrano Recipe

Chef-solo Repository

Server Bootsrapping

More about custom CSS for RT4

Ruby 2 test drive

ruby-1.9.3-p374

ruby-2.0.0-p0

Nagira v0.2.5 release

Background parser

Partial ActiveSupport

Flexible configuration

Nagira is gem now

It's UNIX my dear Watson

Meet Sherlock OS

Documentation

Source code

Memorizable password generation

New tricks and questions with Jekyll

Tagging and tag cloud

How to make tags in pages work?

Sources

Syntax highlight with Pygments

Nagios Checks Aggregation

Nagios Checks Aggregation

Computer Rack Prower Monitoring

Possible uses

See more

Monitoring Applications with RESTful API

Applications with HTTP RESTful API

Nagios For Couchbase

System Architecture