It seems to me that Badiou is himself providing the criticism or response, specifically to theorists like Heidegger and Deleuze. I'm not sure anyone has published any major criticisms of his arguments. In a way, he's attempting to rescue the idea of objectivity, to rehabilitate truth and the notion of the subject, from the ontological criticisms set forth by the theorists who came before. It's a breath of fresh air for those who think of Deleuze as a "dead-end", not necessarily to be read as a contradiction of that work.
The specific criticisms, as I understand them, would probably go something as follows:
Badiou continuously ignores and even refuses to recognize the "suppleness" of power, and its effect on the discourse of alterity. He says that the association of pathos with these forms of alterity is both disruptive and unnecessary, and even attempts to remove alterity and pathos from the equation entirely through the use of mathematical axioms, but this is impossible. Despite the claims made in his works, he is still haunted by an "ineradicable remainder of pathos". Here, you can read the entire earlier works of Emmanuel Levinas (to which Badiou is responding) as themselves indictments of Badiou.
Slavoj Žižek is also a critic in some senses (though he possesses an extreme amount of respect for Badiou's thought, describing him as a "giant who walks among us" and likening him to Plato or Hegel), with the primary point of divergence being Žižek's allegiance to Kant and Hegel, contrasted with Badiou's rejection of both. Žižek is very much on the side of "finitude", whereas Badiou attempts to overcome the very problem of finitude. In particular, though Badiou accuses the philosopher's "denial of enjoyment" as making her vulnerable to asceticism, the philosopher can in turn accuse the psychoanalyst (Badiou) as being transfixed by his own enjoyment, particularly since Badiou's argument is that contemporary culture exists almost entirely "under the emblem of enjoyment". Specifically, Bruno Bosteels makes this observation in an article he contributed to Alain Badiou: Key Concepts, edited by Bartlett and Clemens. (Although Bosteels is attempting to clarify and synergize the works of the two theorists, rather than strictly offering a criticism of Badiou.)
Gilles Deleuze (and, to a lesser extent, his good buddy, Félix Guattari) is probably Badiou's primary antagonist. Although both interpret philosophy as involving the thinking of the multiplicity, Deleuze stubbornly refuses to give in to Badiou's willingness to cede the thinking of enjoyment to the psychoanalyst. The primary author to consider here is Deleuze's student and contemporary,Eric Alliez. Suggested readings might include his essay "Badiou: the Grace of the Universal" and his contributions to the above-suggested title, Alain Badiou: Key Concepts.
Giorgio Agamben also opposes Badiou in his insistence upon the "irreducible remainder". In essence, he argues that the ethic of truth proposed by Badiou actually requires the "occlusion of the remainder", and Agamben's entire understanding of philosophical discourse rests on the potentiality of the remainder, primarily through his central concept of "bare life", which can be interpreted as life which is resting on the very verge of death.