Monday, December 23, 2013

Unix/Linux log script outputs to a file

In any scripts that print something out, the simplest way to capture those printed stuffs is to append with

>> logfilename.log 2>&1 

All the outputs (stdout, stderr) will be redirect to file logfilename.log. To learn this in details, search for something like "unix stdout redirection"

Friday, October 4, 2013

Installing Ruby: cannot load such file -- [ openssl | zlib ]

If you like me, installed Ruby by compiling from source code manually, and found that this 2 errors when trying to install bundler

  • cannot load such file -- openssl
  • cannot load such file -- zlib
You missed configure make to compile with those 2 libraries. Follow these steps to fix this issue.
  1. Remove the installed Ruby with make clean
  2. Install libssl-dev with your OS's package manager of choice. E.g. apt-get install libssl-dev
  3. Install zlib1g-dev with your OS's package manager of choice. E.g. apt-get install zlib1g-dev
  4. Config make file to include openssl by go to ext/openssl and run ruby extconf.rb
  5. Config make file to include zlib by go to ext/zlib and run ruby extconf.rb
  6. Go back to ruby source code directory run make && make install
  7. You should be able to successfully run gem install bundler
You can also do the same thing for readline. Install libreadline-dev and run extconf.rb in ext/readline

Tested with ruby 2.0.0p247
Credit:

Wednesday, August 7, 2013

Download file from S3 via HTTP

You need to have s3cmd installed. This command should return all your buckets
s3cmd ls 
First make the file you want to download be public accessible with command setacl
s3cmd setacl --acl-public s3://bucket-name/path-to-file/filname
 Then you can download from this path
http://bucket-name.s3.amazonaws.com/path-to-file/filename
(You can download with many connection in parallel with axel)

Don't forget to set permission back to private with
 s3cmd setacl --acl-private s3://bucket-name/path-to-file/filname
Credit:  a comment in s3tool.org

Monday, August 5, 2013

FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

This error message from chef is not very useful
FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
I took me a few hours to dig in to chef source code to figure out what's really happening. It turned out for me that chef doesn't like ~ (home) that I used in path declarations in the solo.rb file.

As I skimmed through the code, I have a feeling that any errors that occurs during fork process will cause this error message which does not explain anything. If your cause is not the same as me, unfortunately your only option is to go digging through chef source code. :(

Gem: Chef
Version: 11.6.0

Sunday, May 19, 2013

RubyMine can't find test_helper

I have been an RSpec user since I started working on Ruby and Rails. I usually run a spec file, which reflect the production code file that I'm currently working on, inside RubyMine. So far it has been working fine out of the box.

As a lot more people start talking about Minitest, I want to give it a try. I was starting a new project today. It was a perfect chance to try out Minitest. I expected it to work perfectly out of the box with RubyMine, but apparently it is not.

I got this error when I tried running my test as usual.
`require': cannot load such file -- test_helper (LoadError)
Basically, it says that Ruby can't find test_helper file that is required in the top of my Minitest test file. I knew that it's LOAD_PATH issue but I didn't actually know how to config it properly since I was spoiled by out of the box RSpec integration.

I did googling around by no luck. There's only a page show how to config Minitest report.

I did try to switch back to RSpec just to see whether it's still working fine, and it is. I looked at the runner command that's generated by RubyMine when we execute the spec. There's nothing special from what it generates for Minitest. It seems like RSpec does some work figuring out the path for us. That's why it works out of the box.

So I aim toward finding a way to manually config properly the test running path. Finally, I figured it out with the clue from this comment.

  • First go to Run > Edit Configurations.
  • On the left panel, select Defaults and Ruby
  • Put the absolute path to your Rails project into Working directory: field
  • Append field Ruby arguments with -Itest (tell Ruby to look into test directory)
  • If you had run the test before, you need to remove all the configurations under Ruby section (outside Defaults) to force them to load the new configuration
  • And enjoy the Minitest experiences! (Should work with TestUnit too)
There's one potential problem with this solution. If you do run any non-test Ruby files, they will always load test directory which you don't want them to. Personally I've never do that so this setup is fine for me.

Saturday, May 11, 2013

[Ruby] Run multiple MiniTest files in command line without other setup required (Rake, ...)

MiniTest is a great, simple build-in test framework for Ruby. We can use immediately without anymore setup beside installing Ruby. But there's an annoying limitation when starting a project that we can only run one file at a time via command line. Try to execute many files with "ruby *" doesn't help.

Here's the simplest way to run multiple minitest files I can think of
ruby -e "Dir.glob('**/*_spec.rb').each { |f| require File.expand_path(f) }"
Change suffix _spec to _test for test-unit style file.

Tuesday, May 7, 2013

Wednesday, May 1, 2013

[Blog] Great use of Null Object pattern

Last weekend, I watched a talk Code Smells: Your Refactoring Cheat Codes by John Pignata from MountainWest RubyConf 2013. I was really enjoyed with it. One of the technique that I really like is his use of Null Object pattern. Even though, I know this pattern for long time, but in the situation he was in, I would not be able to come up with the solution like that.

Almost at the end of the talk, his code came to the state that he has jobs and PushDaemon that pushing jobs to the worker. Because a job returned from factory can be nil, he needs to check and allow only non-nil job to be pushed. This is a well-known use case for Null Object Pattern. He modified factory to  return NullJob instead of nil. Now his PushDaemon doesn't need to check for nil anymore.

The code was cleaner without nil checking but pushing null job to worker is not so efficient. How can we avoid pushing NullJob to the worker without checking for nil? That was the time he showed a great technique. Flipping push method (<<) in PushDaemon which was send to worker list to send to job with the opposite direction (>>) and pass worker list as an argument instead. Then implements >> for all the jobs to push itself to worker list except for NullJob. >> of NullJob does nothing. At the end, everything fall cleanly and efficiently.

I highly recommend you to check out his talk for other great refactoring techniques yourself from the link beginning of this blog or on his blog directly.

Saturday, April 20, 2013

pg_dump: Error message from server: ERROR: canceling statement due to conflict with recovery

I ran to this error the other day when I tried to export a dump file from a slave Postgres database.

cpg_dump: Dumping the contents of table "[table_name]" failed: PQgetResult() failed.
pg_dump: Error message from server: ERROR:  canceling statement due to conflict with recovery
DETAIL:  User query might have needed to see row versions that must be removed.
pg_dump: The command was: COPY public.[table_name] ([comma-separated column names]) TO stdout;
I turns out that we can't execute long query on Hot Standby mode server. Right now, there's no perfect solution to this problem but there are some couples of workarounds.

Mailing list related to this issue: Hot Standby - ERROR: canceling statement due to conflict with recovery

Thursday, April 18, 2013

[RSpec] Refer to let value that has the same name in outer scope

There is a situation in RSpec when you want to redefine let value in the inner scope base on value from the outer scope. In RSpec 2.13, you can do that by referring outer scope value with super() (parens are required)

Credit: RSpec 2.13 is released!

Tuesday, April 16, 2013

[Watched] Go at Google

http://www.infoq.com/presentations/Go-Google

Interesting points in Go that I learned from this presentation

  • You can not have unused import statement. Makes code cleaner and faster compilation time.
  • Favor small duplication over including big dependency to use only small part of it
  • No implements declaration required for implementing an interface. Safety from static typing and flexibility from ad-hoc type.
  • Gofix - Cool tool for changing the code to fix language version incompatibility issue.

Friday, March 29, 2013

[Fixed][VIM] E354: Invalid register name: '*'

This error occurs when you or some of your vim plugins try to access register '*' which is a system clipboard. In default Mac OSX vim, it is compiled without system clipboard support enable. You can check this by run command :version inside vim. In the result, you will see -clipboard. Disable is -, enable is +.

To solve this issue, you need to compile vim with +clipboard yourself. I suggest OSX users to use brew to help this process. Out-of-the-box brew compiles vim with +clipboard. To do this run
brew install vim
Then you might need to create a new symbolic link from your vi command to point to the newly installed vim.
First you need to know the path of your new vim. Run
brew info vim
You should see something similar to /usr/local/Cellar/vim/7.3.875 in the result which means your vim binary file is at /usr/local/Cellar/vim/7.3.875/bin/vim. To create a new symbolic link for vi, first you need to remove the existing link
sudo rm /usr/bin/vi
Then you can create a new link.
sudo ln -s /usr/local/Cellar/vim/7.3.875/bin/vim /usr/bin/vi

Sunday, March 3, 2013

PostgreSQL: Listen to remote connection


By default postgres does not allow accessing via ip address, you can only access with localhost. If you try to access with ip address, you will get the error message like this: (assume that 10.0.1.11 = localhost)
$ psql -h 10.0.1.11 dbname
psql: could not connect to server: Connection refused
Is the server running on host "10.0.1.11" and accepting
TCP/IP connections on port 5432?
To allow connection via ip address, you need to add that ip address to listen_addresses config. Here's how to do that:
  1. Open file postgresql.conf (the location of the file depend on OS and installed place)
  2. Look for listen_addresses
  3. Uncomment the line
  4. Add ipaddress that you want postgres to listen on inside the same single quote with 'localhost'. Like this listen_addresses = 'localhost,10.0.1.11'
  5. Restart postgres server (again depend on your OS on how to restart service)
  6. Now, you should be able to access it with "-h ipaddress". Such as psql -h 10.0.1.11 dbname

Saturday, February 23, 2013

SSHFS - Mount remote directory through gateway machine


  1. Expose target target server ssh port with ssh local port forwarding
    $ ssh -C -N -L localhost:local_port:target:target_ssh_port gateway_user@gateway
  2. Mount with sshfs through the port just exposed
    $ sshfs target_user@localhost:/target_directory /mount_point -o port=local_port
Example:
$ ssh -C -N -L localhost:22222:tap-machine:22 tapvpn@company-vpn-host.org
$ sshfs tap@localhost:~/work /mnt/work -o port=22222



Thursday, February 21, 2013

Execute one line command on IRB or Rails's console through Unix shell

Basic Unix approach - echo and pipe:
$ echo "<command>" | irb
For example,
$ echo "1 + 1" | irb
1 + 1
2
You can change irb to rails console to execute that command in the Rails environment.

Thursday, February 7, 2013

PostgreSQL - export/import schema only

Export

Use -s to specify schema only
pg_dump <src_database_name> -s > schema.sql
(Optional) Use -t to specify table. For more than 1 table just add more -t.
 pg_dump <src_database_name> -s -t <table1_name> -t <table2_name> > schema.sql

Import
psql <dest_database_name> -f schema.sql


Credit:

Allow other people to publish gem to RubyGems

To do that, you need to add them as an owner of that gem with gem owner command.

This is an example
gem owner gem_name -a foo@bar.com
Credit: How to add more 'owners' to a gem in RubyGem

Thursday, January 31, 2013

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

I ran to this issue couples of week ago
Installing oj (2.0.2) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /Users/viki/.rvm/rubies/ruby-1.9.3-p194/bin/ruby extconf.rb 
>>>>> Creating Makefile for ruby version 1.9.3 on x86_64-darwin11.4.0 <<<<<
creating Makefile

make
compiling cache.c
compiling cache8.c
compiling dump.c
compiling fast.c
compiling load.c
compiling oj.c
compiling saj.c
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal 2 (use -v to see invocation)
clang: note: diagnostic msg: Please submit a bug report to http://developer.apple.com/bugreporter/ and include command line arguments and all diagnostic information.
clang: note: diagnostic msg: Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/qw/xj8ns4cn09jfv1x5qq1mhn7m0000gn/T/saj-4CFmfe.i
clang: note: diagnostic msg: /var/folders/qw/xj8ns4cn09jfv1x5qq1mhn7m0000gn/T/saj-4CFmfe.sh
make: *** [saj.o] Error 254


Gem files will remain installed in /Users/viki/.rvm/gems/ruby-1.9.3-p194@raynor/gems/oj-2.0.2 for inspection.
Results logged to /Users/viki/.rvm/gems/ruby-1.9.3-p194@raynor/gems/oj-2.0.2/ext/oj/gem_make.out

I could find the way to fix it. It was quite frustrated at that moment because the only way to get passed this error to start our web server again is to downgrade oj gem.

But eventually, I found the way to fix it. As I understand, this error causes by bug in llvm (Apple's gcc that comes with XCode). Here is the steps of how I fixed the issue.

First you need to install gcc4.2 with brew

brew update
brew tap homebrew/dupes
brew install apple-gcc42
Then reinstall your Ruby with this command (I use RVM)

CC=/usr/local/Cellar/apple-gcc42/4.2.1-5666.3/bin/gcc-4.2 rvm install 1.9.3-pXXX
After this you should be able to install oj or the other gems as usual.

I composed solution from many links but sorry I forgot to take note of those.



Config number of file descriptors on Linux

Check file descriptor limit for a process. Vary depending on owner of the process $ cat /proc/{process-id}/limits Check current number of...