So I'm not sure how but I managed to brick the atmega328p chip. Found an error in PCB - usbasp socket didn't have GND connected anywhere on the board. I guess the only reason it worked during programming is because I used power from usb2ftdi adapter from the same PC. But when I tried to play with being powered from phone battery and tried to program the chip at the same time - it went silent. I tried many things (like external oscilator), but nothing helped :(

I decided to unsolder the chip, after 3 days of different unsoldering tricks this little bastard gave up. He broke 3 legs and probably his neck too, but I got him out :)

Lesson learned: for testing/first board it's better to use a socket for a chip :)