I’m interested in both Japanese and Chinese and I can speak and understand both to some level.
After 3 years of living in Japan without formal Japanese courses, and a year of living in China, I realized that Japanese is harder to sound natural and native than Chinese. I still sound like a retard when I speak in Japanese. For Chinese, there are many instances when I talk to the locals and they don't get my tones so get flustered but when I found my bearing I can converse much more fluidly.
The most difficult thing I encouter when learning Japanese is that the lessons on textbooks and schools are retarded compared to what is actually spoken. Have you ever heard anyone say the following?
私はあなたにえんぴつをあげる。(Watashi ha anata ni empitsu wo ageru)
I swear that you will never, ever hear this sentence in your entire life. In fact, 90% of the time 私 (Watashi) and （あなた）Anata are not used. Compared this to Chinese where 我 (wo) and 你 (Ni) are commonly heard from a regular conversation. The lack of subjects usage in Japanese is a hurdle which until now I find difficult to overcome.
Some people have the wrong idea that since Mandarin uses 4 tones (+ 1 neutral tone) then it is much harder than Japanese which is mostly flat. But the truth is, the tonal foundation is an advantage in the long run. I believe that languages are better learned through the ears than through the eyes (a proof of which is that deaf people cannot make themselves understood), and hence devoting a long time in learning the tones and the music of Chinese produces the long-term result.
My last complain on Japanese is that there are so many readings for a certain character compared to Chinese which has at most 2 readings for some of its characters. Take for example:
行 - Could be read as kou, kyou, gyou, an, i..ku, yu..ku, okona..u/i in Japanese
行 - Only Xing2 and Hang2 in Chinese
Anyway, I still hope that one day I can speak both languages fluently and even be able to translate from Japanese to Chinese and vice versa.