Calories do matter. It just so happens that when we make bad choices of food we take in much larger amounts of calories. Sure you can say certain macronutrients are better for building muscle/losing fat but in the end a caloric deficit will cause you to lose weight. I'm eating much more volume of food right now but still losing weight. It's hard to pack on calories eating just chicken, brown rice, and veggies. I could get 1 big mac and take in more calories than my entire day of chicken/rice.