Unix/Linux log script outputs to a file

In any scripts that print something out, the simplest way to capture those printed stuffs is to append with

>> logfilename.log 2>&1 

All the outputs (stdout, stderr) will be redirect to file logfilename.log. To learn this in details, search for something like "unix stdout redirection"

Installing Ruby: cannot load such file -- [ openssl | zlib ]

If you like me, installed Ruby by compiling from source code manually, and found that this 2 errors when trying to install bundler

• cannot load such file -- openssl
• cannot load such file -- zlib

You missed configure make to compile with those 2 libraries. Follow these steps to fix this issue.

1. Remove the installed Ruby with make clean
2. Install libssl-dev with your OS's package manager of choice. E.g. apt-get install libssl-dev
3. Install zlib1g-dev with your OS's package manager of choice. E.g. apt-get install zlib1g-dev
4. Config make file to include openssl by go to ext/openssl and run ruby extconf.rb
5. Config make file to include zlib by go to ext/zlib and run ruby extconf.rb
6. Go back to ruby source code directory run make && make install
7. You should be able to successfully run gem install bundler

You can also do the same thing for readline. Install libreadline-dev and run extconf.rb in ext/readline

Tested with ruby 2.0.0p247

Credit:

• http://askubuntu.com/questions/261914/install-ruby-2-0-with-openssl-and-readline-support
• http://stackoverflow.com/questions/769496/ubuntu-noob-rails-install-fails-on-zlib

Download file from S3 via HTTP

You need to have s3cmd installed. This command should return all your buckets

s3cmd ls 

First make the file you want to download be public accessible with command setacl

s3cmd setacl --acl-public s3://bucket-name/path-to-file/filname

 Then you can download from this path

http://bucket-name.s3.amazonaws.com/path-to-file/filename

(You can download with many connection in parallel with axel)

Don't forget to set permission back to private with

 s3cmd setacl --acl-private s3://bucket-name/path-to-file/filname

Credit:  a comment in s3tool.org

FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

This error message from chef is not very useful

FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

I took me a few hours to dig in to chef source code to figure out what's really happening. It turned out for me that chef doesn't like ~ (home) that I used in path declarations in the solo.rb file.

As I skimmed through the code, I have a feeling that any errors that occurs during fork process will cause this error message which does not explain anything. If your cause is not the same as me, unfortunately your only option is to go digging through chef source code. :(

Gem: Chef
Version: 11.6.0

RubyMine can't find test_helper

I have been an RSpec user since I started working on Ruby and Rails. I usually run a spec file, which reflect the production code file that I'm currently working on, inside RubyMine. So far it has been working fine out of the box.

As a lot more people start talking about Minitest, I want to give it a try. I was starting a new project today. It was a perfect chance to try out Minitest. I expected it to work perfectly out of the box with RubyMine, but apparently it is not.

I got this error when I tried running my test as usual.

`require': cannot load such file -- test_helper (LoadError)

Basically, it says that Ruby can't find test_helper file that is required in the top of my Minitest test file. I knew that it's LOAD_PATH issue but I didn't actually know how to config it properly since I was spoiled by out of the box RSpec integration.

I did googling around by no luck. There's only a page show how to config Minitest report.

I did try to switch back to RSpec just to see whether it's still working fine, and it is. I looked at the runner command that's generated by RubyMine when we execute the spec. There's nothing special from what it generates for Minitest. It seems like RSpec does some work figuring out the path for us. That's why it works out of the box.

So I aim toward finding a way to manually config properly the test running path. Finally, I figured it out with the clue from this comment.

• First go to Run > Edit Configurations.
• On the left panel, select Defaults and Ruby
• Put the absolute path to your Rails project into Working directory: field
• Append field Ruby arguments with -Itest (tell Ruby to look into test directory)
• If you had run the test before, you need to remove all the configurations under Ruby section (outside Defaults) to force them to load the new configuration
• And enjoy the Minitest experiences! (Should work with TestUnit too)

There's one potential problem with this solution. If you do run any non-test Ruby files, they will always load test directory which you don't want them to. Personally I've never do that so this setup is fine for me.

[Ruby] Run multiple MiniTest files in command line without other setup required (Rake, ...)

MiniTest is a great, simple build-in test framework for Ruby. We can use immediately without anymore setup beside installing Ruby. But there's an annoying limitation when starting a project that we can only run one file at a time via command line. Try to execute many files with "ruby *" doesn't help.

Here's the simplest way to run multiple minitest files I can think of

ruby -e "Dir.glob('**/*_spec.rb').each { |f| require File.expand_path(f) }"

Change suffix _spec to _test for test-unit style file.

[Ubuntu] Remove Password Prompt For sudo

• Open a terminal
• Run sudo visudo
• Add line 
• <username> ALL=NOPASSWD: ALL
• For example, david ALL=NOPASSWD: ALL
• Save and quit
• Done

From: https://help.ubuntu.com/community/RootSudo#Remove_Password_Prompt_For_sudo

[Blog] Great use of Null Object pattern

Last weekend, I watched a talk Code Smells: Your Refactoring Cheat Codes by John Pignata from MountainWest RubyConf 2013. I was really enjoyed with it. One of the technique that I really like is his use of Null Object pattern. Even though, I know this pattern for long time, but in the situation he was in, I would not be able to come up with the solution like that.

Almost at the end of the talk, his code came to the state that he has jobs and PushDaemon that pushing jobs to the worker. Because a job returned from factory can be nil, he needs to check and allow only non-nil job to be pushed. This is a well-known use case for Null Object Pattern. He modified factory to  return NullJob instead of nil. Now his PushDaemon doesn't need to check for nil anymore.

The code was cleaner without nil checking but pushing null job to worker is not so efficient. How can we avoid pushing NullJob to the worker without checking for nil? That was the time he showed a great technique. Flipping push method (<<) in PushDaemon which was send to worker list to send to job with the opposite direction (>>) and pass worker list as an argument instead. Then implements >> for all the jobs to push itself to worker list except for NullJob. >> of NullJob does nothing. At the end, everything fall cleanly and efficiently.

I highly recommend you to check out his talk for other great refactoring techniques yourself from the link beginning of this blog or on his blog directly.

pg_dump: Error message from server: ERROR: canceling statement due to conflict with recovery

I ran to this error the other day when I tried to export a dump file from a slave Postgres database.

cpg_dump: Dumping the contents of table "[table_name]" failed: PQgetResult() failed.
pg_dump: Error message from server: ERROR:  canceling statement due to conflict with recovery
DETAIL:  User query might have needed to see row versions that must be removed.
pg_dump: The command was: COPY public.[table_name] ([comma-separated column names]) TO stdout;

I turns out that we can't execute long query on Hot Standby mode server. Right now, there's no perfect solution to this problem but there are some couples of workarounds.

• Make a dump from master.
• Set max_standby_archive_delay and max_standby_streaming_delay to be bigger that amount of time it needs to export a dump. This will make slave catching up master version slower. It's not likely to work for a big database.
• Pause sync, make a dump and the resume sync. You can pause sync with function pg_xlog_replay_pause() and resume with function pg_xlog_replay_resume(). See more details on this 2 functions from 9.24. System Administration Functions and New Replication and Recovery Features in PostgreSQL 9.1. Note that we should not pause it too long (> 24 hours), otherwise I might not be able to resume.

Mailing list related to this issue: Hot Standby - ERROR: canceling statement due to conflict with recovery

[RSpec] Refer to let value that has the same name in outer scope

There is a situation in RSpec when you want to redefine let value in the inner scope base on value from the outer scope. In RSpec 2.13, you can do that by referring outer scope value with super() (parens are required)

Credit: RSpec 2.13 is released!

[Watched] Go at Google

http://www.infoq.com/presentations/Go-Google

Interesting points in Go that I learned from this presentation

• You can not have unused import statement. Makes code cleaner and faster compilation time.
• Favor small duplication over including big dependency to use only small part of it
• No implements declaration required for implementing an interface. Safety from static typing and flexibility from ad-hoc type.
• Gofix - Cool tool for changing the code to fix language version incompatibility issue.

[Fixed][VIM] E354: Invalid register name: '*'

This error occurs when you or some of your vim plugins try to access register '*' which is a system clipboard. In default Mac OSX vim, it is compiled without system clipboard support enable. You can check this by run command :version inside vim. In the result, you will see -clipboard. Disable is -, enable is +.

To solve this issue, you need to compile vim with +clipboard yourself. I suggest OSX users to use brew to help this process. Out-of-the-box brew compiles vim with +clipboard. To do this run

brew install vim

Then you might need to create a new symbolic link from your vi command to point to the newly installed vim.
First you need to know the path of your new vim. Run

brew info vim

You should see something similar to /usr/local/Cellar/vim/7.3.875 in the result which means your vim binary file is at /usr/local/Cellar/vim/7.3.875/bin/vim. To create a new symbolic link for vi, first you need to remove the existing link

sudo rm /usr/bin/vi

Then you can create a new link.

sudo ln -s /usr/local/Cellar/vim/7.3.875/bin/vim /usr/bin/vi

PostgreSQL: Listen to remote connection

By default postgres does not allow accessing via ip address, you can only access with localhost. If you try to access with ip address, you will get the error message like this: (assume that 10.0.1.11 = localhost)

$ psql -h 10.0.1.11 dbname
psql: could not connect to server: Connection refused
Is the server running on host "10.0.1.11" and accepting
TCP/IP connections on port 5432?

To allow connection via ip address, you need to add that ip address to listen_addresses config. Here's how to do that:

1. Open file postgresql.conf (the location of the file depend on OS and installed place)
2. Look for listen_addresses
3. Uncomment the line
4. Add ipaddress that you want postgres to listen on inside the same single quote with 'localhost'. Like this listen_addresses = 'localhost,10.0.1.11'
5. Restart postgres server (again depend on your OS on how to restart service)
6. Now, you should be able to access it with "-h ipaddress". Such as psql -h 10.0.1.11 dbname

SSHFS - Mount remote directory through gateway machine

1. Expose target target server ssh port with ssh local port forwarding
   $ ssh -C -N -L localhost:local_port:target:target_ssh_port gateway_user@gateway
2. Mount with sshfs through the port just exposed
   $ sshfs target_user@localhost:/target_directory /mount_point -o port=local_port

Example:

$ ssh -C -N -L localhost:22222:tap-machine:22 tapvpn@company-vpn-host.org

$ sshfs tap@localhost:~/work /mnt/work -o port=22222

Credit: Mounting Filesystems with SSH Forwarding Across a Gateway

Execute one line command on IRB or Rails's console through Unix shell

Basic Unix approach - echo and pipe:

$ echo "<command>" | irb

For example,

$ echo "1 + 1" | irb

1 + 1

2

You can change irb to rails console to execute that command in the Rails environment.

PostgreSQL - export/import schema only

Export

Use -s to specify schema only

pg_dump <src_database_name> -s > schema.sql

(Optional) Use -t to specify table. For more than 1 table just add more -t.

 pg_dump <src_database_name> -s -t <table1_name> -t <table2_name> > schema.sql

Import

psql <dest_database_name> -f schema.sql

Credit:

• How to Import / Export in Postgres Database ?
• How to import existing *.sql files in Postgresql 8.4?
• pg_dump

Allow other people to publish gem to RubyGems

To do that, you need to add them as an owner of that gem with gem owner command.

This is an example

gem owner gem_name -a foo@bar.com

Credit: How to add more 'owners' to a gem in RubyGem

[Solved] Could not open a connection to your authentication agent.

Run this command

eval `ssh-agent`

Credit:  http://www.worldofnubcraft.com/1926/could-not-open-a-connection-to-your-authentication-agent-ssh-add-error/

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

I ran to this issue couples of week ago

Installing oj (2.0.2) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /Users/viki/.rvm/rubies/ruby-1.9.3-p194/bin/ruby extconf.rb 
>>>>> Creating Makefile for ruby version 1.9.3 on x86_64-darwin11.4.0 <<<<<
creating Makefile

make
compiling cache.c
compiling cache8.c
compiling dump.c
compiling fast.c
compiling load.c
compiling oj.c
compiling saj.c
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal 2 (use -v to see invocation)
clang: note: diagnostic msg: Please submit a bug report to http://developer.apple.com/bugreporter/ and include command line arguments and all diagnostic information.
clang: note: diagnostic msg: Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/qw/xj8ns4cn09jfv1x5qq1mhn7m0000gn/T/saj-4CFmfe.i
clang: note: diagnostic msg: /var/folders/qw/xj8ns4cn09jfv1x5qq1mhn7m0000gn/T/saj-4CFmfe.sh
make: *** [saj.o] Error 254


Gem files will remain installed in /Users/viki/.rvm/gems/ruby-1.9.3-p194@raynor/gems/oj-2.0.2 for inspection.
Results logged to /Users/viki/.rvm/gems/ruby-1.9.3-p194@raynor/gems/oj-2.0.2/ext/oj/gem_make.out

I could find the way to fix it. It was quite frustrated at that moment because the only way to get passed this error to start our web server again is to downgrade oj gem.

But eventually, I found the way to fix it. As I understand, this error causes by bug in llvm (Apple's gcc that comes with XCode). Here is the steps of how I fixed the issue.

First you need to install gcc4.2 with brew

brew update

brew tap homebrew/dupes

brew install apple-gcc42

Then reinstall your Ruby with this command (I use RVM)

CC=/usr/local/Cellar/apple-gcc42/4.2.1-5666.3/bin/gcc-4.2 rvm install 1.9.3-pXXX

After this you should be able to install oj or the other gems as usual.

I composed solution from many links but sorry I forgot to take note of those.